Building a Media Player #5: The Server-side Node.js Code

Building a Media Player #5: The Server-side Node.js Code

PAUL LEWIS: Hello, and welcome
to yet another developer diary. So to this point,
I’ve shown prototypes and explained a
little bit of theory. But finally– finally, but
finally, we actually get on to talking about
what’s been going on in the actual application code. And in this entry, I think I’m
going to mostly concentrate on the server side. So firstly, I suppose
I need to acknowledge that a lot of this code
really came from Surma. If you’ve not seen our
Supercharged episode where he talked about
server side rendering, definitely watch that. It was a live stream where
he took about an hour or so and built a relatively
straightforward, Handlebars-based, Express-based,
with caching headers, server side rendering
thingy setup. And I guess, yeah,
a lot of this is– the ideas, anyway– have
been picked up from there about how to do some of
the caching and so on. And it was also
Surma who told me about Express sub-apps
or Express apps, that you can actually
have lots of them. But more of that in
a moment because, well, there’s code to look at. So let’s have a dive in
and see what’s there. So this is actually
the server as it runs today, which
largely involves getting a copy of
Express, making an app to wrap everything else. And I’m running all my
stuff on the flex containers in Google’s app engine. So what you can do
there is you can boot up a container which runs Node. It’s great. It’s like that. And then it wraps it in the auto
scaling, load balancing stuff. It’s really quite handy
for a lot of this. But it patches in the port
as part of the environment variable. So we basically pick that up. And that’s what we run on,
whatever it tells us to run on. Beyond that, there’s
some middleware. And I’ll come back to
some of this in a moment. I have also got
passport enabled, which is a way to
do a [INAUDIBLE] if you need to do it. And I’m not sure if
I’m going to or not, whether I’ll have an
admin area or whether I’ll have a place where people
can actually log in so they can see their own– what they’ve rated or whatever. I’m not sure yet. But I’ve put it in here just
for now in case I need it. But the interesting ones
actually kick in below. So there is an app
here for static. So anything that begins
/static, I hand off to this app, which is under Apps, Statics. Let me have a quick
look under there. And this is actually relatively
straightforward, this one. It’s basically going
to call express.static with a path and the ops. And the ops is that we want
anything that comes from /static to have an
age of one year on it. So it will expire in one
year, not a moment sooner, apparently. But that’s the idea. We’re going to say anything
that begins /static, grab it from there. In this case, it actually
goes to the Client folder, which is up here,
and grabs it from there. We’ll come back to that, I
suspect, in a little bit. So anything that begins /static,
which is images, the manifest, so on and so forth, go
and grab those from there. The authentication one– I’m
not going to get into that right now. That’s the stuff that
would use Passport. Because I’m not sure if it’s
actually going to survive, if I actually need
it or not, there isn’t a lot of point in
me showing that just yet. If I do do anything
that requires it, I’ll come back and explain that. It’s mostly placeholder, which
is the same with the admin. It’s just me testing
the auth stuff. I’ll tell you what. We’ll jump over the
service worker one, and we’ll come back
to it in a bit. Most of the app
is actually served through this dynamic
one, this dynamic app. So let me show you
around that a little bit. So this is not dissimilar
to the static one, except it’s way, way bigger. It’s its own Express app,
which I call Dynamic. I’m using Adaro. Adaro? This is the problem with
reading names online. You never know how
to pronounce them. And then you have to
say them out loud. And there’s a good
chance that you’re going to look like an idiot. I don’t need much help
to look like an idiot, just so you know. Glad we had this chat. Anyway, I’m using– I’m going to say Adaro, I think. And if it’s wrong, oh, well. And what that does
is it wraps Dust JS. So Dust JS is an alternative
to Handlebars or Dot JS– any of those, really. I wanted to give it a go
because it supports promises and streams, as I understand
it, so that you can– in theory, at least, if you want to– make your templates–
build them async. So if you need to
do something which requires a little
bit of async work, then you’re not blocking
forever while that happens. All the same, it seemed
like an interesting thing to go and play with. So that’s what I’m doing. In order to make
it work in Node, though, you need to
wrap it around Dust JS. And that’s what Adaro is. Then we have a bunch of
setup stuff, really, which gets me things that
I need about reading my video library, information
about the videos that I’ve got, information about the
actual package itself. Because I use things like the
version number in the package JSON to stamp certain files to
make sure that that all works. I have a bunch of helpers, which
again, we can come back to, which are– they’re helpers that help
me with the templates that Dust JS is going to create. Finally, we have a
little bit of setup here for the actual
default view, like what the title
is going to be, the page, what scripts
we need, and so on. Then we step into the
actual config of the engine, where we can say create dust
engine, set up the views to use the view path,
and also, anything that comes from the dynamic
side is going to be HTML. And therefore, I don’t
want it to cache at all. So if you remember
the stuff in /static, it had a one-year expiry. The stuff from dynamic, I
want it to expire immediately. I never want it to be cached. Because what we’re going to do– I call it cache and hash. You could do cache and
grab, hash and grab, cache and hash and grab, any of
the above, whatever you like. But I call it
caching and hashing. And this is something that
I definitely got from Surma. So we’ll talk about
that more in a moment. But the idea is, first of
all, anything that’s HTML, don’t cache that. Anything that is from /static,
definitely cache that, and cache it for a year. Right. So let’s talk about– this is one of the
roots in the home page. We grab the Default
View Options. We add in the things that
we need to add specifically for that view like, which
is the featured video, which are the newest video, any
things that we want in lines, like the JavaScript and the CSS,
any particular scripts that we want to add to that. And then we finally just
call .render against it with the template that we’ve
got, which is home. And that’s it. It will basically render
that out and send that back to the client. Works just great. OK, let’s talk more about
the cache and hash stuff. The idea is that if you
have an HTML file with paths to, say, the manifest or
the JavaScript or the CSS, and any of those
change, you need to flag that to the browser. Because you’ve said anything
that comes from /static, like your JavaScript, like
your CSS, like your manifest, they should be
cached for a year. So what you need to do is you
need to do something probably to the file name. That’s what most of us do. We put a revision
into the file name so that you can cache
that revision for a year. But if I change the HTML,
or if I change the files, the HTML will get a
new revision in there for that particular file. So let me show you
what I actually mean. Let me bring up the browser. Wrong browser. Let me switch across
to the correct browser. So I’m running the site
on 8080 at the moment. Ta-da. And let me show you. Here’s a great one here. Yeah, this is ideal,
the manifest here. You can see that we’ve got
this– manifest dot, and then this long string here, dot JSON. Well, how are we doing that? Let me show you here. This is one of the middlewares
that I mentioned earlier. Or is it helper? It’s a helper. Sorry. It’s one of the helpers, which
we’ve registered up here, this one here, the hash helper. What it does is it
gets handed a path. And what it does is
it opens up that file. You can see here, load
the file contents. And then it creates a hash
from the file contents by making a SHA-256, writing
the file data in there, and asking for a hex dump
out the other side, which is 64 hexadecimal digits long. And we use that, and we
replace the file name. So if it was– you can see here. Let me actually bring– let me grab– let
me show you this. In the header, there we are. This is what it
actually looks like. So I said, look, get the
dist client manifest.json. And it will grab that file,
look at the file contents, create the hash for that. And then it will spit
back out an updated path that looks like this, all that. There we are. Now, that’s good, and it’s bad. It’s good because that
means this is unique. So when the manifest changes,
this hash will also change. Brilliant. But it’s also on /static, which
means that it will be cached for a year. So that’s all good. The only thing now is when
we actually make the request, this file doesn’t
actually exist in /static. I mean,
/staticmanifest.json exists. But manifest dot all that
dot JSON doesn’t exist. So what we have
on the other side is a little bit of middleware
that just simply removes the hash. If it gets given a request that
has 64 hexadecimal characters right next to one
another, A to F, 0 to 9, then it replaces those
with an empty string, and then passes
straight through. So it’s a completely
silent bit of middleware that just says if you get a
URL with that long hexadecimal string, remove it. And we’ll now ask for
/static/manifest.json, which means when we
come back to the server, we’ve asked for the file. We’re going to get
whatever the server has as the most recent value. So that’s a way of keeping
that all up to date. Last little bit I’m going to
show you on the server side today is around what
I’ve been thinking and what I’ve been doing to
do with the service worker. There was– there was a service
worker app that was dedicated just to the service worker. Fairly similar, in some
ways, to the dynamic 1. But all it does is it totally
controls the service worker only. That’s the only thing that
it’s actually interested in. So not dissimilar
in terms of setup. We’ve got all the Adaro stuff,
and we’ve got default dust options here, and so on. But what we do is– you see that I’ve got this
package reader, .getversion. And all that’s doing
is it’s opening up the package JSON, which is– it’s currently set to 1.0.0. And inside here– inside– well, I lost it. There it is. Inside here, you can see that
I render this service worker template, and I pass
through version. So when we look over here at– I believe it’s over here. Nope. That’s the built
version, I think. Oh, no, it’s here. You can see that we have
the curly braces version. So that means that
if I want to bump the version of the
whole site, and I want to invalidate
my service worker, I can do that just fine. Because what I’ll do is
I’ll bump the version in the package.JSON. And I’ll deploy a new version. And then on the server
side, we’ll generate a new– well, that version
number will be passed through to the service worker. So when we next request
the service worker, we’ll get 1.0.1, for example. That’ll be byte
different to 1.0.0. And then we’ll run the whole
install activate process. So my means– I’ve used it before
a bunch of times now. I did it in Chrome Dev Summit. I’ve done it in– I think it was Voice
Memos I did it in, possibly even Guitar
Tuner as well. A bunch of the stuff
that I’ve built is really convenient to just
go, bump the package version, and then send that out. And that will cause your service
worker stuff to be updated. And the other thing that it also
does, or it should do– yes, it does– is it makes sure that my
service worker is never cached. This is a super important thing. Even though,
technically, I guess, this might– no, it
doesn’t live in static. But even if Express wanted
to cache the service worker for a day or a week,
we really don’t want that. Service workers,
you really want them to cache for no amount of time. And the reason is
we want to be sure that if we update
our service worker, that we definitely come back– the people who are
connected to us definitely come back and get it. As it happens, even if you
set a service worker to cache for more than a day,
the browser will always come back after one day. I believe that’s in the spec. It does mean, though, that if
you set the caching headers to be like a year,
you’re going to have to wait at least a day
before any changes you make to the service worker get
passed down to anybody who’s already got it installed. So strong recommendation
for you there– make sure that
your service worker has some kind of
no caching headers, very much like the HTML, so
that if you make a change, it can get deployed instantly,
as in as soon as somebody hits your site, they’ll
go, oh, I need to get a new service worker. And if they can, they’ll
download and activate and install, and all the rest
of it, your new service worker. So there you go. All the code, as I mentioned
previously, is up on GitHub. I will link to it
in the notes below. So that’s the service
side of things. Next up, I think we should
have a little bit of a chat about maybe more video stuff. Because that is
what we’re building. Don’t forget, you can
subscribe to the channel. Don’t forget, you can
leave comments below. Loving reading all the comments. Thank you so much
for all of those. It really does make
all this worthwhile because it’s so nice
to see what you think and also to get your
questions as well. So if you’ve got questions,
definitely pop those in. And I’ll try to either
respond to them in line or in the next
video that I record. Too-da-loo. Hey, folks. Thanks for watching. Don’t forget that there
is more content that you can find kind of over here-ish. And if you want to subscribe,
there’s probably a button– I don’t know, maybe there,
maybe somewhere around there. Click that if you’ve
not done not that. Brilliant.


20 thoughts on “Building a Media Player #5: The Server-side Node.js Code”

  • Penyemak Imbas Chrome – Google 4.5/5 bintang (6,927,327 penilaian)
    Penyemak imbas Google Chrome yang anda sukai pada desktop membolehkan anda menyemak imbas dengan …

  • Alessandro Menduni says:

    Would love to learn more about the deployment of these projects! Also, would you integrate the app with a CMS if it were supposed to be sent to production?

  • As I understand it assets gets overwritten on update and the server implicitly redirects all filenames containing a hash to the file without the hash string (without checking validity).

    Are there any advantages of doing the remove hash dance and not just use links like /static/styles.css?_h=123abcdef (only adding the hash string to the queryparams in the build step)?

  • I really love this series format. One question: With the revision hash, is it throwing the file's content through the crypto function on every request? If so: is this bad or a concern for performance? Would caching it somehow be an option?

  • Paul, you are the most entertaining developer I've yet to come across (for years), your spontaneous puns always gives me a good laugh no matter the day.

  • What about serving videos, from my experience dabbling with something similar in Go I've always thought that's a possible pain point since serving large hi res videos by loading all of it into memory isn't a good option. Does express static do that for you or some other way?

  • Colin Richardson says:

    Anyone know what IDE this is? I originally thought it was Atom. and when when I went to use it, I realised, I was wrong..

  • Hi,

    Would this file hashing/revision solution also work if we opted by marking these static files as "static" in app.yaml (so they can be served by appengine cdn directly) ? Not sure about the real/practical benefits of that though…


Leave a Reply

Your email address will not be published. Required fields are marked *