Even Facebook struggles: Zuck's titanic database upgrade hits numerous legacy software bergs

Facebook has had all sorts of no fun trying to migrate from MySQL 5.6 to version 8.0.

A post from the social network’s engineering team reveals that Facebook's last MySQL upgrade – to version 5.6 – took "more than a year".

Moving to version 8.0 has taken "a few years so far".

Some of the reasons for the slow rollout will sound familiar – such as continued work on legacy software, even as new infrastructure was being implemented.

Facebook had custom code – over 1700 patches in its in-house branch of MySQL 5.6. Even as it ported them to 8.0, the company was adding more custom features to version 5.6 that also needed to be dragged into the future.

• Facebook gardening group triumphs over slapdash Zuck censorbots
• Facebook pulls plug on mind-reading neural interface that restored a user's speech
• Pakistan bans TikTok, for the fourth time

Other parts of Facebook legacy code wouldn't work with MySQL 8.0, and therefore "required a deprecation and migration path forward".

"Upgrading from 5.6 to 8.0 skipped over 5.7 entirely. APIs that were active in 5.6 would have been deprecated in 5.7 and possibly removed in 8.0, requiring us to update any application using the now-removed APIs," wrote Facebook software engineer Herman Lee and production engineering manager Pradeep Nayak.

Complicating matters further was Facebook's use of the MyRocks project, an effort that lets MySQL use RocksDB as its storage backend, and which Facebook was developing during its migration to MySQL 5.6.

Jumping from version 5.6 to 8,0 – and skipping version 5.7 – meant Facebook could not upgrade servers in place, so had to use logical dump and restore to build a new server.

"However, for very large mysqld instances, this can take many days on a live production server and this fragile process will likely be interrupted before it can complete. For these large instances, we had to modify our backup and restore systems to handle the rebuild," Lee and Nayak wrote.

Facebook tracked the porting process of its custom patches using … wait for it … spreadsheets – and recorded discrepancies as they went.

"Discrepancies on porting status would inevitably arise due to the large number of patches we needed to sift through and these notes helped us resolve them," the pair wrote.

For migration, Facebook hatched a plan to create secondary instances of MySQL databases, then copy data from version 5.6. Once that was done, the secondary version 8.0 instance would be copied to the primary.

To get that done at Facebook scale, "we needed to build new software infrastructure", the post states matter-of-factly.

We had to modify our backup and restore systems to handle the rebuild

Other issues that cropped up included increased memory usage, file format incompatibilities, and conflicts between newly adopted keywords and "popular table column names and aliases used in application queries".

The tone used by the post's authors suggests the project has made steady progress, with glitches to be overcome rather than setbacks that derailed progress or left project timelines in tatters.

The post ends by saying the apps Facebook has moved are benefiting from their new database environment.

"Overall, the new version greatly expands on what we can do with MySQL @ Facebook," Lee and Nayak conclude.

Whether that's good for the rest of us is, as ever, the question Facebook struggles to answer. ®