MangaDex, a scanlation platform with tens of millions of monthly visitors, has developed an innovative solution to help satiate its users' thirst for content. The site's newly open sourced MangaDex@Home peer-to-peer project allows users to volunteer use of their PCs or servers to help ease the pressure on the site's cache servers.
Serving a reported tens of millions of visitors each month, unofficial ‘scanlation’ platform MangaDex is a sizeable operation.
This manga community site offers translated copies of manga comics to a worldwide audience, something that comes with its own set of complications and hurdles to overcome.
Earlier this year the platform permanently lost access to Cloudflare and for a while had to change domain.
After these issues had been put behind them, the operators of the site began to experience problems with bandwidth too. This month they revealed that during the coronavirus / COVID-19 lockdown, traffic increased by 15%. Then, a month later, MangaRock finally threw in the towel, an event that increased traffic to MangaDex by another 15%.
Adding insult to injury, the site learned that one of its providers could no longer cache its image archive traffic, something which led to “dismal loading times for old chapters.” At the same time, however, the site revealed an extremely interesting innovation called MangaDex@Home.
“MangaDex@Home is a P2P (peer-to-peer) system where users will be able to volunteer the usage of either their personal computers or servers to act as cache server nodes to alleviate the stress on our own cache servers. Over time, we envisage that the majority, if not all, of the older chapters will be served by MangaDex@Home,” the site announced.
For people familiar with the mechanics of BitTorrent distribution, the MangaDex@Home system will certainly ring some bells. Rather than content being hosted centrally, those running the dedicated MangaDex client (participation is voluntary) will host content on their own machines, acting as servers from where regular users can access content, thereby distributing bandwidth stresses and costs.
“You will be hosting a client that acts as a P2P system for older chapters,” MangaDex revealed. “Basically, your machine will act as a server where a tiny portion of older MangaDex chapters will be stored and when a reader wants to read an older chapter, it will be ‘fetched’ from your machine and served to the reader.”
At the moment, MangaDex is recruiting volunteers with specific resources at their disposal, including a minimum network speed of 80Mbps up/down, at least 40GB of dedicated storage space, and a promise that the PC or server will be online 24/7.
Earlier this month and just after launch, the site reported a combined output of 6650Mbps and over 18TB of cache space but noted that more capacity would be needed in the future. According to the most recent update, uptake has been impressive.
“The amount of people volunteering servers to participate in MangaDex@Home is greater than we could have imagined,” the platform announced. “Currently all users and all guests are set to receive images from the MangaDex@Home network and the initiative itself has brought our cache server traffic down to acceptable levels.”
That the MangaDex community has responded in this fashion shouldn’t come as a surprise. While they all share a love for manga, many have also been involved in another distributed computing project.
The Folding@Home project uses idle computing resources to help combat diseases such as cancer, ALS, Parkinson’s, Huntington’s, Influenza, and more recently, COVID-19. In May, the MangaDex team broke into the top 500 contributors.
This week MangaDex announced that the MangaDex@Home project is now open-source, meaning that anyone can contribute to its development moving forward.
