Filtering Duplicate Messages in Twitch Chat
I recently built this toy command line interface (CLI) app for fun. You can see a higher resolution video of it in action by clicking the image below.
The video above is a snapshot of the CLI in action during the middle of a speed run of the Nintendo 64 game The Legend of Zelda: Ocarina of Time, hosted on Twitch.tv. It shows the filtering in action, in the middle of a burst of the
WutFace spam, which you can see in the lower left of the above video.
For the uninitiated, Twitch.tv is a streaming video website that allows anyone to share live video with viewers over the internet, with twitch. The most popular use case by a wide margin for streaming players playing video games, although it is also sometimes used for sharing live music, where the video streamer might be playing an instrument (or instruments, if it’s a band.
Several thousand people can be watching the same video stream at the same time — it’s not uncommon to see streams that have 100,000 simultaneous viewers. Since viewers who watch the same video stream are also put into the same chat room (there’s no chat room splitting that happens automatically), the chat room messages can become incredibly difficult to parse.
To (sort of) solve this problem, I wrote a simple CLI Twitch/IRC chat client that supports the ability to filter out duplicate messages within a sliding time window in realtime.
Although this was built specifically for Twitch, it works with any chat that supports IRC as a protocol.
The fully functional IRC client was written in Golang. The choice of Golang was purely out of my desire to learn the language. If I was to build something beyond for toy usage, I’d probably write it as a Chrome extension, since native Twitch chat is in the browser, and it’s where users are accustomed to chatting.
blessed-contrib are limited in scope.
There are the following widgets in this client:
1) Message Rate Monitor (Rate of messages over sliding window)
2) Message Stats Monitor (Min, Max, Avg messages over sliding window)
3) Duplicate Message Aggregator (List of messages sorted by number of occurrences over sliding window)
The trickiest implementation detail was the duplicate message aggregator. It involved maintaining a sorted list of
(message, count) pairs in realtime as messages arrived, and then decrementing / removing the message from the sorted list of (message, count) pairs, while also preventing concurrent updates from malforming the sorted list counts.
Despite the complexity, Golang made this process quite easy. Concurrent updates were handled by serializing all updates through a queueing channel, and goroutines with
time.Sleep(duration) were used to decrement counts.
Further Thoughts on Golang
Golang was a bit of a pain at times — the lack of generics in the language shows. I had to duplicate methods like
int types, since the builtin
math.Max only works with
float64. Also, the extra overhead of thinking about when to pass a pointer to a struct vs just the struct got in the way of thinking about the higher level functionality of the application.
On the flip side, I spent almost no time learning Golang and was able to build something functional almost immediately. This hasn’t been the case with other languages I’ve learned, such as
Scala. The expediency experienced with Golang comes from two things — firstly, the amazing tooling that has gone into the language ecosystem, and secondly, the sheer simplicity of the language and lack of language features.
I wouldn’t use Golang as my first language choice for prototyping a product, but for system level programming, network intensive applications, or performance critical backends, Golang seems like the perfect choice for development. I’ll stick to Python, Scala, or TypeScript for prototyping products quickly, and use Golang when the extra performance is necessary.
Using the Client
The source code and instructions for usage of the project described earlier can be found at my github page here. All that is needed is a golang installation and a Twitch account.