Downloading Web Videos

Trying To Download Videos From Sites Without A Plug-In.

Introduction

I'm writing this post because I'm fascinated at the different ways people play videos on their site. So when I'm bored, I usually start poking around people's web code to figure out ways to download videos. Of course, you can always use a plug-in or other tool to do so. Those tools usually gather the streaming info from the Networks data and just piece it back together. I might show how to do that from within the browser, but for now, I'm going to start off by mainly focusing on searching for direct links to videos.

Inspecting is video elements is a great way to learn how people load videos on their site. I don't condone using it for anything illegal though. Think before you act.

So far, I've found ways to download videos these sites:

  • PBS
  • Vimeo
  • Facebook

And found ways to download videos from sites using these video services:

  • Wistia
  • BrightCove

I assume you're kind of familiar with your browser's developer tools, otherwise, you'll probably be pretty confused by this tutorial. I mostly use Chrome. To open them on Mac, you press option + command + i, at the same time.

After going through each of these, you'll have better understanding of:

  1. The many HTMl elements used to display videos.
  2. How video configuration data is moved around.
  3. How video data is sent and received.
  4. How you can really capture anything using the Networks tab.
  5. What the limitations of downloading videos are.

I hope to add sites like Twitter & Youtube eventually. This tutorial will never have sites like Netflix & Hulu though because they use EME (Encrypted Media Extensions). So, even if we did download the video (probably very possible), you wouldn't be able to watch it without a special key. Besides the point, it's not something people should be able to do anyways.

PBS

PBS is relatively simple. When you watch a video on their site, you get a url that looks like this: http://www.pbs.org/video/3000009137/

Anytime you see something that looks like an identity key (3000009137), you should search the DOM for it. In this case, open the Elements tab, click anywhere in the elements area, open the text search (Cmd + F on Mac) and search for the key. Scroll through the hits, and one of the hits should look like this javascript code:

copy
PBS.videoData = {
    'alternate_encoding': {
        'url': 'http://urs.pbs.org/redirect/05f1051670944a4b9bb15b120335a595/',
        'eeid': 'mp4-baseline-16x9'
    },
    'recommended_encoding': {
        'url': 'http://urs.pbs.org/redirect/daab766ea9a44ca7a92b70938f5fd5d4/',
        'eeid': 'hls-800k-16x9'
    },
    'image_url': 'http://image.pbs.org/video-assets/pbs/school-inc/265558/images/mezzanine_571.jpg.fit.980x551.jpg',
    'closed_captions_url': 'http://ga.video.cdn.pbs.org/captions/school-inc/4d09782f-e19a-41a5-8961-c0694d38e390/captions/764383_Encoded.dfxp',
    'program': {
        'slug': 'school-inc',
        'producer': 'PBS',
        'title': 'School Inc.',
        'is_excluded_from_dfp':  false
    },
    'duration': 3291,
    'chapters': chapters,
    'title': 'The Price of Excellence (Episode One)',
    'contentID': 2365991343,
    'videoType': 'Episode',
    'is_mvod':  false,
    'airdate': ''
};

The first url will take you to the direct link of the video. Just copy and paste it into a different tab, and then you can download it using your browser's video download button or by right clicking and selecting Download Video.


Wistia

Wistia also uses an identity key in its url. Searching that in the DOM will lead you to javascript that looks like this:

copy
window.theMediaEmbed = Wistia.embed("efo38fj319", {
  accountKey: "wistia-production_3356",
  container: "wistia_video_embed_container",
  doNotTrack: false,
  uuid: "wistia_video_embed",
  dontFit: "false",
  videoFoam: "true"
});

It appears that they've created a unique object and appended it to the window. So go to the console and print the object with console.log(window.theMediaEmbed);

That prints a large object to search. We want to look for a property that would probably described where the video is coming from. In this case, what we're looking for is data.media.assets

In that nested property, you'll find an array of objects. Open each one, and you'll see that they contain data on loading the same video, but with different encodings. Each object has a url property with a url to its video. Go to that url in a separate window, and you can download the video.


Vimeo

Here's a interesting one that also uses an identity key. Search the DOM for "<video" to find the video element. There should just be one. One of its attributes will be "src", but this one uses a blob url. You can't go to a blob url directly because they actually point to local data (which is stored on your computer).

copy
 
<video preload="" src="blob:https://vimeo.com/d58bde3c-bbba-452f-8289-f5d436a30b8e"></video>

Here's the neat trick. Replace the blob url in the video tag with the url of the page, which should be like https://vimeo.com/212323382. That will force the video to reload, except this time, instead of stitching together a blob url, it will just load the remote url into the video element. If that doesn't work, try using the pag url with video added to the path like this: https://vimeo.com/video/212323382.

The src attribute should now contain something like this: https://fpdl.vimeocdn.com/vimeo-prod-skyfire-std-us/01/2464/8/212323382/729105832.mp4?token=1492210592-0xa708dc6d1000077a144e011ef75bf93323f5b1d8

copy
 
<video preload="" src="https://fpdl.vimeocdn.com/vimeo-prod-skyfire-std-us/01/2464/8/212323382/729105832.mp4?token=1492210592-0xa708dc6d1000077a144e011ef75bf93323f5b1d8"></video>

Go to that url in a separate window, and you can download the video.


Facebook

This one follows closely to Vimeo, but requires something extra. Most likely, you want to download a video that you saw in your news feed. When you right click on the video, you should see an option to "Show video url". Click that, and a little pop up on top of the video should show up with a link inside. Copy that link and open in a separate window.

Search the DOM on this page for the video tag just like you did for Vimeo & replace the blob url with the page url. The video element will now dissapear.

The video has been replaced by an anchor link. On the page, it just looks like the video thumbnail before being played. Click the video thumbnail to play the video, which will load the video as an embed element.

Now search the DOM for "<embed" to find the embed element which is now hosting the video. One of its attributes should be flashvars which has a bunch of JSON data in it. Grab that data. It is currently encoded. To decode it, open the console, and use the decodeURIComponent() function to decode the JSON. Make sure you wrap the data in double quotes when you insert it into the function. It should look something like below.

copy
 
decodeURIComponent("params=%7B%22auto_hd%22%3Afalse%2C%22autoplay_reason%22%3A%22unknown%22%2C%22default_hd%22%3Atrue%2C%22disable_native_controls%22%3Atrue%2C%22inline_player%22%3Afalse%2C%22pixel_ratio%22%3A1%2C%22preload%22%3Afalse%2C%22start_muted%22%3Afalse%2C%22rtmp_stage_video%22%3Atrue%2C%22rtmp_buffer%22%3Afalse%2C%22rtmp_buffer_time%22%3A0%2C%22rtmp_buffer_time_max%22%3A0%2C%22video_data%22%3A%7B%22progressive%22%3A%7B%22is_hds%22%3Afalse%2C%22video_id%22%3A%221482546135130316%22%2C%22is_live_stream%22%3Afalse%2C%22rotation%22%3A0%2C%22is_facecast_audio%22%3Afalse%2C%22sd_src_no_ratelimit%22%3A%22https%3A%5C%2F%5C%2Fvideo.fapa1-2.fna.fbcdn.net%5C%2Fv%5C%2Ft42.1790-2%5C%2F17930263_750899061754914_8048250619389018112_n.mp4%3Fefg%3DeyJ2ZW5jb2RlX3RhZyI6InN2ZV9zZCJ9%26oh%3Da48287ddc451b4da020a0252d16eb2b2%26oe%3D58F17F94%22%2C%22hd_src_no_ratelimit%22%3A%22https%3A%5C%2F%5C%2Fvideo.fapa1-2.fna.fbcdn.net%5C%2Fv%5C%2Ft43.1792-2%5C%2F17879334_1895441397391761_1308002508693569536_n.mp4%3Fefg%3DeyJ2ZW5jb2RlX3RhZyI6InN2ZV9oZCJ9%26oh%3D14656645a0a7da9bc8610b2113283905%26oe%3D58F1842E%22%2C%22aspect_ratio%22%3A1.7777777777778%2C%22hd_src%22%3A%22https%3A%5C%2F%5C%2Fvideo.fapa1-2.fna.fbcdn.net%5C%2Fv%5C%2Ft43.1792-2%5C%2F17879334_1895441397391761_1308002508693569536_n.mp4%3Fefg%3DeyJybHIiOjE1MDAsInJsYSI6NDA2MywidmVuY29kZV90YWciOiJzdmVfaGQifQ%5Cu00253D%5Cu00253D%26rl%3D1500%26vabr%3D757%26oh%3D14656645a0a7da9bc8610b2113283905%26oe%3D58F1842E%22%2C%22sd_src%22%3A%22https%3A%5C%2F%5C%2Fvideo.fapa1-2.fna.fbcdn.net%5C%2Fv%5C%2Ft42.1790-2%5C%2F17930263_750899061754914_8048250619389018112_n.mp4%3Fefg%3DeyJybHIiOjMwNywicmxhIjo5MTcsInZlbmNvZGVfdGFnIjoic3ZlX3NkIn0%5Cu00253D%26rl%3D307%26vabr%3D171%26oh%3Da48287ddc451b4da020a0252d16eb2b2%26oe%3D58F17F94%22%2C%22hd_tag%22%3A%22sve_hd%22%2C%22sd_tag%22%3A%22sve_sd%22%2C%22stream_type%22%3A%22progressive%22%2C%22live_routing_token%22%3A%22%22%2C%22projection%22%3A%22flat%22%2C%22subtitles_src%22%3Anull%2C%22dash_manifest%22%3Anull%2C%22dash_prefetched_representation_ids%22%3Anull%2C%22player_version_overwrite%22%3A%22pleasantville%22%2C%22spherical_config%22%3A%5B%5D%2C%22is_spherical%22%3Afalse%2C%22original_height%22%3A720%2C%22original_width%22%3A1280%7D%2C%22hls%22%3Anull%7D%2C%22video_data_preference%22%3A%7B%221%22%3A%7B%22is_hds%22%3Afalse%2C%22video_id%22%3A%221482546135130316%22%2C%22is_live_stream%22%3Afalse%2C%22rotation%22%3A0%2C%22is_facecast_audio%22%3Afalse%2C%22sd_src_no_ratelimit%22%3A%22https%3A%5C%2F%5C%2Fvideo.fapa1-2.fna.fbcdn.net%5C%2Fv%5C%2Ft42.1790-2%5C%2F17930263_750899061754914_8048250619389018112_n.mp4%3Fefg%3DeyJ2ZW5jb2RlX3RhZyI6InN2ZV9zZCJ9%26oh%3Da48287ddc451b4da020a0252d16eb2b2%26oe%3D58F17F94%22%2C%22hd_src_no_ratelimit%22%3A%22https%3A%5C%2F%5C%2Fvideo.fapa1-2.fna.fbcdn.net%5C%2Fv%5C%2Ft43.1792-2%5C%2F17879334_1895441397391761_1308002508693569536_n.mp4%3Fefg%3DeyJ2ZW5jb2RlX3RhZyI6InN2ZV9oZCJ9%26oh%3D14656645a0a7da9bc8610b2113283905%26oe%3D58F1842E%22%2C%22aspect_ratio%22%3A1.7777777777778%2C%22hd_src%22%3A%22https%3A%5C%2F%5C%2Fvideo.fapa1-2.fna.fbcdn.net%5C%2Fv%5C%2Ft43.1792-2%5C%2F17879334_1895441397391761_1308002508693569536_n.mp4%3Fefg%3DeyJybHIiOjE1MDAsInJsYSI6NDA2MywidmVuY29kZV90YWciOiJzdmVfaGQifQ%5Cu00253D%5Cu00253D%26rl%3D1500%26vabr%3D757%26oh%3D14656645a0a7da9bc8610b2113283905%26oe%3D58F1842E%22%2C%22sd_src%22%3A%22https%3A%5C%2F%5C%2Fvideo.fapa1-2.fna.fbcdn.net%5C%2Fv%5C%2Ft42.1790-2%5C%2F17930263_750899061754914_8048250619389018112_n.mp4%3Fefg%3DeyJybHIiOjMwNywicmxhIjo5MTcsInZlbmNvZGVfdGFnIjoic3ZlX3NkIn0%5Cu00253D%26rl%3D307%26vabr%3D171%26oh%3Da48287ddc451b4da020a0252d16eb2b2%26oe%3D58F17F94%22%2C%22hd_tag%22%3A%22sve_hd%22%2C%22sd_tag%22%3A%22sve_sd%22%2C%22stream_type%22%3A%22progressive%22%2C%22live_routing_token%22%3A%22%22%2C%22projection%22%3A%22flat%22%2C%22subtitles_src%22%3Anull%2C%22dash_manifest%22%3Anull%2C%22dash_prefetched_representation_ids%22%3Anull%2C%22player_version_overwrite%22%3A%22pleasantville%22%2C%22spherical_config%22%3A%5B%5D%2C%22is_spherical%22%3Afalse%2C%22original_height%22%3A720%2C%22original_width%22%3A1280%7D%2C%222%22%3Anull%7D%2C%22show_captions_default%22%3Afalse%2C%22persistent_volume%22%3Atrue%2C%22hide_controls_when_finished%22%3Afalse%2C%22rtmp_start_playing_non_zero_stream_time%22%3Atrue%2C%22rtmp_improve_playback_config%22%3Afalse%2C%22rtmp_no_stream_reported_time%22%3Afalse%2C%22rtmp_start_time_fix%22%3Atrue%2C%22buffer_length%22%3A0.1%7D&width=1280&height=720&user=1690520169&log=no&div_id=id_58f155891392c7c61008958&swf_id=swf_id_58f155891392c7c61008958&browser=Chrome+57.0.2987.133&tracking_domain=https%3A%2F%2Fpixel.facebook.com&post_form_id=");

Press return, and the function should print out the decoded JSON. There are several links in this JSON that lead to different qualities of the video. Grab any one of them (I think "hd_src_no_ratelimit" leads to the highest quality version). We need to change one more thing. Everytime you see a forward slash "/", it is being escaped by a back slash "\". Remove every back slash you see. I think there are 5.

Finally, you should end up with a url that looks something like this: https://video.fapa1-2.fna.fbcdn.net/v/t43.1792-2/17879334_1895441397391761_1308002508693569536_n.mp4?efg=eyJ2ZW5jb2RlX3RhZyI6InN2ZV9oZCJ9&oh=14656645a0a7da9bc8610b2113283905&oe=58F1842E

Go to that url in a separate browser, and you can download the video.


BrightCove

BrightCove is another video service like Wistia. Sometimes it's easiest to use the Networks tab, but it doesn't load unless it's already open. So first, open the developer's console, open the Networks tab, and go to: https://www.brightcove.com/en/player

The name of the request we're looking for is different depending on the site. It's supposed to refer to the video ID we're looking for, which is typically a random number like: 4592378961001. Though, I've seen videos with actual names, not just numerical IDs.

Once you find the right request, you'll know, because the "Preview" tab should show the body of the request with JSON in it like this:

copy
{description: "Create and publish branded players that are automatically optimized for all devices.",…}

Open up the JSON using those triangle icons on the left of it. Open up the sources key. That will display a list of objects where each one contains info on the same video with different encodings, just like we found with Wistia! Choose the version you want to download and grab the link from the src key. Make you chose one where the link starts with "http". Copy and paste that link in a new tab, and download the video using the download button.


...

We'll be diving more and more into just the Networks tab when I post the following hacks...