Moving Recordings to a New Backend
My current MythTV machine has been expanded as far as it will go, so its time to upgrade. :-) I've built a more powerful server and I want to transition everything to it. I need to move all my data at once to the new server because I don't have enough space to run two servers permanently. And I have 1TB of data, so efficient transport will be an important factor.
Data Transfer
Initially, I attempted to move the TV recordings with a graphical SFTP client. But after several minutes, I realized my transfers were maxing out at about 2.2MBps. That's less than 20% of the capacity of my 100MBps network. I don't have anything else running, so what's limiting my download speed? I decided to investigate. Here are the results of my tests:
| Transfer Method | Example Command | Speed |
|---|---|---|
| SFTP with hostname | ''sftp me@my.machine.com:/video/file.nuv . |2.2MBps | |SCP with hostname |scp me@my.machine.com:/video/file.nuv .'' | 2.2MBps |
| SFTP with IP address | ''sftp me@192.168.1.111:/video/file.nuv . '' | 11.2MBps |
| SCP with IP address | scp me@192.168.1.111:/video/file.nuv . | 11.1MBps |
| SCP with IP & blowfish | scp -c blowfish me@192.168.1.111:/video/file.nuv . | 11.1MBps |
| SCP with IP & compression | scp -C me@192.168.1.111:/video/file.nuv . | 4.4MBps |
I read somewhere that using a less CPU-intensive encryption algorithm like blowfish would make my SCP transfers faster, but it didn't help. And turning on SSH compression (the -C) was reported to improve download speeds on slow connections. I tried it on my fast network and it actually made things worse.
So here's what I learned: using a hostname rather than an IP address made a big performance difference. It probably has something to do with my router configuration. I'll ignore that for now...the point is that I got an 11.1MBps transfer rate, which is 91% of the theoretical max of 12.5MBps. The difference is due to protocol overhead.
Data Synch
My MythTV backend is constantly recording new shows. Hence, the set of files I copied to my new server become stale within hours. I need a way to synchronize my new machine with the existing one until I can complete my conversion. Enter rsync.
cd /my/video/folder
rsync -avz me@my.machine.com:/source/folder/ .
I'm forced to enter my password with this command because I specified my user ID, but at least this got my files transferred.