Codewars Leaderboard & Calendar
Been collecting data for a while, was about time I did something with it all!
First I needed to sanitize my data, as currently is being scraped every three hours, and for two reasons wanted to squash this to a daily data file. One was the fact that sometimes my scraper under-fetched users while paginating, which means there's a change a user would be missing, but squashing 8 scrapings into a single dataset would greatly reduce the chance of a user being missed. The other reason would be to protect a little user privacy - seeing at which 3-hour period someone gained honor can tell you quite a bit about a user.
Quality of Life Improvements
Before that I had a few little quality of life improvements, one being the automatic fetching of a REMEMBER_USER_TOKEN using only a users Email and Password, allowing people to use these scripts without having to find that cookie value themselves.
The next being some step-by-step guides to how to do various tasks the scripts allow, from downloading solutions, to creating git repositories.
Additionally began tracking solution votes, so users can find out all the katas they forgot to vote on and obtain their additional point of honor!
The userscript also got some improvements - first the removal of duplicate rows added by Codewars, and secondly the ability to filter which clans users are shown from.
The last improvement was actually making my script compatable with those using Node v14, which was reported to me by Twitch Chat, and I was able to do by only replacing all my .at(-1) usages with .slice(-1)[0].
Next.js Frontend
I decided to go with a Next.js frontend for this project, for no other reason then I knew I'd need some backend processing, yet already possessed the data I needed to process as flatfiles.
So I started off with a basic user-lookup feature, allowing the entering of a username and the fetching of information from both the official Codewars API, and my function - providing additional information such as the user profile image URL, and all the additional user stats displayed on the homepage not captured by the API.
Next was the leaderboard - which I wanted to have two inputs, a start and end date, allowing for one to see the position & honor changes between those two dates.
The start date would take the data from the start of the day, and to match, the end date would take the data from the end of the day, so if the dates were the same there would still be a change reported.
Calendar
This was the primary feature of the site, allowing users to see who gained the most honor in various degrees:
- Days
- Week
- Day of Week
- Month
The frontend wasn't super simple, I decided to create my own calendar UI, so I generated all the days between the day of first data from my backend to the current date, generating an extra row to display who has gained the most honor on those days of the weeks, and an extra column to show who gained the most honor for those weeks. Finally I added the first of that period's avatar to the day, and implemented username filtering so you can easily see what rank you were on the calendar for all these periods.
The generation of the calendar was something I could probably improve, as my approach was to consume all the data for each period that would need to be calcuated, doing this for all days, then weeks, and finally months, due to me writing this so inefficiently I implement caching when the requested dates never changed, and only regenerated - while using previously generated data - when the underlying data changed.
While this does work, and only takes a whopping five minutes to generate new data, there are various other improvements I want to make to this site first.