3 Ways to Speed up Raspberry Pi's File Transfer

The Raspberry Pi 4 is the fastest of all Pi. It brings new interfaces to the Pi family: The USB 3.0 port, and gigabit ethernet. Now, people won't have to be constrained by the crappy USB 2.0 interface of the previous Raspberry Pi generations. The Raspberry Pi Compute Module 4 is even better. A whole PCIe 2.0 port? That's like 5Gbps of transfer speed! That is enough to enjoy the lightning speed of NVMe SSDs.

"Oh my goodness, I can't wait to abuse its 5GT/s data link!"I thought.

No! Of course there is a catch. Raspberry Pi 4's CPU does not have hardware-acceleration of common encryption ciphers (wink AES wink). The Raspberry Pi foundation somehow decided that hardware-based encryption is of no importance, even though its competitors have that years ago!

So anything remotely related to encryption will be slow. To name a few:

• Downloading a file from the Internet (Most websites use HTTPS)
• Anything related to torrents
• Encrypting hard drives
• ...and yes, SSH and SFTP

Fortunately, hope is not all lost. It is still possible to saturate the 1Gbps ethernet link while downloading your favorite movies. Similarly, you can reduce the CPU load on your puny Raspberry Pi while transferring large files.

Use a 64-bit OS for your Pi

Yes, a 64-bit OS actually speeds up your Raspberry Pi, by a lot. I mean, Raspberry Pi 4's bcm2711 processor supports 64-bit, might as well use it.

If you have enough RAM, that is. It is true that 64-bit systems consume more RAM than their 32-bit counterparts. RAM won't be a problem for most Pi users, as most Pi have 2GB RAM. So if your Pi 4 is the 1GB RAM model, why did you buy it?2GB model costs like the same price as the 1GB model. Moreover, if you have a 4GB or 8GB model, using a 64-bit system enables an application to allocate more than 3GB RAM. So, quite a win-win situation I believe.

Official 64-bit Raspberry Pi OS

64-bit OS historically had very bad support from the Raspberry Pi Foundation. The good news is: Albeit in beta, Raspberry Pi Foundation finally released an official 64-bit Raspberry Pi OS Lite! It is better than 32-bit Raspberry Pi OS in almost every way. Even the system itself takes up less spaces! It can be downloaded here.

Basically, if you are most people, use the 64-bit Raspberry Pi OS. It automagically makes all programs faster, which includes cryptographic libraries.

Testing SFTP speed

Some people might wonder: "I know 64-bit makes applications faster, but how do you know it improves SFTP specifically?" It is actually quite easy to figure it out yourself. Just connect to the Pi itself and try to upload a file!

The procedure is simple:

1. Create a temporary file on the Pi in a ramdisk.
2. On the Pi, connect to itself via SFTP.
3. Upload the temporary file to a ramdisk, basically copying the file through SFTP.

This eliminates factors such as SD card performance, ethernet stability, and USB performance.

And the testing can be done by executing the commands below on your Pi:

dd if=/dev/zero of=/tmp/testfile bs=1M count=512 status=progress; sync
sftp pi@localhost
sftp> put /tmp/testfile /tmp/out.tmp

From my tests, SFTP's transfer speed averages about 55.9 MB/s on a 32-bit system, while averaging about 76.6 MB/s on the 64-bit one.

That is a stunning 37% improvement in upload speed, by switching Pi OS to 64-bit alone!

Overclocking

Yes. Overclocking. If the program is CPU-bound, and you need to squeeze out some extra performance, the obvious choice is to overclock the CPU and make some smoke.

And yes, encryption ciphers are CPU-bound. So let's try overclocking!

Overclocking might damage your Raspberry Pi!

Basically, you only need to insert these two lines into your /boot/config.txt:

over_voltage=2
arm_freq=1750

This config will overclock your Raspberry Pi's CPU from 1.5 GHz to 1.75 GHz. About ~15% increase in frequency.

And the results are pretty good:

Our overclocked Pi yields 84.4 MB/s SFTP transfer speed. A nice 10% increase. If you want to be more overkill, you can overclock the CPU to 2.0 GHz for a theoretical 102.13 MB/s upload speed. That is pretty close to saturating the 1Gbps ethernet!

But what if we want something else?

Use an alternative file transfer protocol

It is known that SFTP is horribly slow.

It is true! For example, SFTP uses chacha20-poly1305 cipher on Raspberry Pi, which is actually really fast. If you benchmark its speed on the Pi with this command

openssl speed -evp chacha20-poly1305

That is some 300 MB/s of raw chacha20-poly1305 speed! So, if there are alternative means such as HTTP download, use it! SFTP is simply way too inefficient for file transfer.

Use an alternative encryption method

We can simply ask SSH/SFTP to use a more Raspberry Pi-friendly cipher. aes-gcm is horribly slow and (somewhat) insecure on devices without hardware AES. Meanwhile, chacha20-poly1305 is the de-facto cipher for mobile devices without AES hardware acceleration.

chacha20-poly1305 should already be the default cipher for Raspberry Pi. It provides enough security and speed.

But if you want more speed, you can use a weaker and potentially insecure cipher like arcfour. Just don't do it.

Conclusion

So yeah, considering Raspberry Pi's raw CPU power, saturating a 1Gbps link is a piece of cake even with encryption (unless you are using SFTP like me). Using a 64-bit system and a moderate overclocking results in 40% faster file transfer in SFTP. This really shows the power of a modern instruction set (aarch64).

Thus, if there are other means to upload/download files, please use them. Even HTTPS download is faster than SFTP.

Hey! It might even be possible to use a PCIe-to-ethernet adapter and saturate a 5Gbps ethernet link! Unfortunately, that is for another time.