Return to Blog Home

Twitch Downloader

 

Twitch Downloader

Yet another twitch Video downloader...

It was brought to my attention that the experience of some users to download segments of Twitch VODs included going to the VOD, creating a clip - first limitation, the length of the clip - then taking this clip to a download site, and downloading the clip.

Now while I personally know how to use utilities such as yt-dlp, there's nothing much more user friendly then a simple website/extension.

In an effort to improve this I started seeing if I could use untwitch.com as a backend to a userscript/extension. The site uses reCAPTCHA to protect it's video-downloading endpoint, so while replicating it's POST request wouldn't be difficult, the reCAPTCHA would not be a simple task, so I decided not to pursue this challenge after passing along my findings.

Productive Diffuse Mode

After not thinking about the prompt for a while, I remembered that yt-dlp itself obtains the HLS playlist and downloads it in it's entirety...and my previous HLS playlist experience reminded me that these playlist are segmented into multiple MPEG-TS files, each with their duration described in near-plaintext.

Proof Of Concept

So entered my first hypothesis: Can load a HLS playlist, and download only certain segments of it - all on the web?

I quickly discovered that HLS playlists are not natively supported, so I threw it into hls.js and made some progress. I was able to grab the video/audio chunks, concatenate them as they buffered, an save the concatenated chunks as two files - one for video and one for audio. I might've continued with this idea has I not ran into a CORS issue upon trying an actual Twitch HLS.

I'm glad I did though, as this made me step back and realize I was over complicating this with trying to play the video and parse the buffered data. Now I'm no MPEG-TS expert, but I remember that it's possible to easily concatenate them together with no recoding needed - at the cost of not being able to seek the final video as they don't contain an index structure.

This lead to my next hypothesis: Can I download the segments of a HLS playlist and concatenate them into a single file?

Of which was quickly proven true, making my proof of concept valid. Next was to develop the actual program to do this - it was either going to be a userscript, or web extension. I decided on making it a userscript as I already had been working on a userscript recently - Twitch User-Markers - and I couldn't think of any reason it needed to be a web extension.

MVP

While I could envision the Twitch Clip creation UI being overladed over the entire video, then after clicking download it would start downloading the segments needed for that section of the video with a beautiful progress bar, I knew with my design experience that would take a very long time. So I set out to just make it work with four main elements:

I did have one challenge that while I knew was possible, I hadn't started working on: obtaining the actual HLS playlist file from a Twitch VOD ID. This had already been proven possible by yt-dlp, thankfully I had issues with yt-dlp in the past and knew how to have it dump it's network requests: --print-traffic

So after dumping the traffic I found a grand total of three requests - the first of which was only obtaining chapters, so easily skippable. The next was using the GQL API as I expected, and was thankfully quite simple, all it took was the Video ID and returned a videoPlaybackAccessToken, which was passed almost verbatim to the next request which return the all the different HLS playlists according to quality. With this discovered as relatively easily doable, nothing was stopping me from starting on the userscript.

Userscript

Since I already had a working userscript specially for Twitch, I simply copied the entire project and spent the next few minutes deleting everything I wouldn't need until I was left with code relating with:

Finally I started developing the new features - first being the UI. I decided on positioning it below the Share button, so after a simple query selector and a few document.createElement()s, I had all the UI components created. Next was to actually develop the HLS fetching and segment filtering code, and like the exiting ID/Video ID Twitch API code, it'll all be cached. After finishing the HLS parsing I only needed to filter down all the files to just the ones that contained the times between the start and end the user desired, which leads to the first tradeoff: the downloaded video can be as much as 20 seconds longer then requested.

This is due to the fact that all the MPEG-TS files are ten seconds in length, so if a user requests a start of 35 and a end of 55, they'll downloaded video will be from 30 to 60. I had already thought of some possible options to this, such as ffmpeg.wasm, but such polishing was for after the MVP was finished.

After dealing with a few parsing errors on my part, it was fully working. You enter a start and end time in DHMS format, hit download, and watch as the <progress> fills up with every segment downloaded, and at the end you're MPEG-TS file is automatically downloaded!

Polishing

At this point I decided to take a break from this userscript, but I had already finished half of the MVP at this point, with only a few things blocking the release:

...and some more features I've still to evaluate the complexity of, to see if they can creep into the MVP scope or not: