Ok, it’s time for some real task for our Raspberry Pi. Today we’ll learn how to configure a command line client for Amazon Glacier and push GBs of data to the cloud. We’ll also configure our Gmail account to send us an email when our RPi is done uploading data. Last, but not least, we will learn how to limit upload rate for RPi so that other devices can still use the internet.
I’ve recently used my RPi for this very task, pushing out 120GB of my photos and backups up to Glacier. It took quite a while on my not-so-good internet connection - I left it running for a couple of weeks. The great thing about RPi is that it’s pretty much inaudible, even with HDD spinning 24/7 which makes it a perfect little server that can run under your desk. Let’s get right to it!
1. Getting up to date
Let’s log in and update system
ssh pi@raspberrypi.lan
sudo apt-get update && sudo apt-get upgrade
2. Install glacier-cmd
We will start by installing git and required python libraries and then install glacier-cmd
sudo apt-get install python-setuptools git
git clone git://github.com/uskudnik/amazon-glacier-cmd-interface.git
cd amazon-glacier-cmd-interface
sudo python setup.py install
3. Configure it
Let’s create a config file for glacier-cmd and fill it in
nano .glacier-cmd
Add the following, replacing your_access_key, your_secret_key and your_aws_region with correct values
[aws]
access_key=your_access_key
secret_key=your_secret_key
[glacier]
region=your_aws_region
logfile=~/.glacier-cmd.log
loglevel=INFO
output=print
Now you should be able to see your vaults by executing
glacier-cmd lsvaults
Success!
4. New vault
Let’s create a new vault for our photos
glacier-cmd mkvault "photos"
You should be able to see a new “photos” vault on the list
glacier-cmd lsvaults
5. Uploading a test file
Ok, now we can try to upload a file. I’m going to upload a file “ocr_pi.png” that I can see in my home directory
glacier-cmd upload --description "ocr_pi.png" photos "ocr_pi.png"
As you can see, I set the description to match the filename. By default, it would be set to a full path to the file, which is something that we don’t want, thus a description parameter.
6. Uploading multiple files
Now we’re going to create a script that will take care of uploading a bunch of files. Navigate to a folder that holds the files that you want to upload. In this example I’m going to upload zip archives. I trust you can figure out how to prepare your files for the upload. I’ve tried to keep every zip file below 500MB, making it easier to upload and also download data in the future, in case I need to access part of it.
Let’s create a folder where we’ll move uploaded files
mkdir uploaded
and a new script inside the folder with
nano upload.sh
and paste the following
find . -name "*.zip" | sort | while read file ; do
echo "Uploading $(basename "$file") to Amazon Glacier."
glacier-cmd upload --description "$(basename "$file")" photos "$file" && mv "$file" "uploaded"
done
Now we can execute the script with
bash upload.sh
It should upload all files one by one, showing progress and rate as it does its thing.
7. Installing screen
All good and well, but we still can’t leave it running on its own, uploading away all the files that we’ve prepared. We could, in theory, use cron job for that, but I personally like to be able to see the progress in real time whenever I want.
We’re going to install screen, a little utility that will let us disconnect from ssh session, while it’s still running and connect back to it at a later time, as if we never left.
sudo apt-get install screen
Now, let’s start a session called simply “pi” within screen with
screen -S pi
You might notice that not much has changed. In fact, everything looks exactly the same. But let’s see what screen will let us do. We will start top and disconnect, then we will try to reconnect and see if top is still running
top
Now press ctrl+a and d after that, you should see information similar to
[detached from 3068.pi]
We can now exit our ssh session with
exit
If everything went well, top is still running on our RPi even though we’re disconnected. Let’s see
ssh pi@raspberrypi.lan
screen -r pi
Boom, top is still running! As you can see, from system’s perspective we’ve never logged out. It’s going to be really useful for glacier-cmd. Just log into your “pi” screen session and execute our bash script as before
bash upload.sh
Now you can disconnect with ctrl+a followed by d and reconnect later to see how the script is doing. Neat, eh?
8. Email notification
I’d also like to be notified when RPi is done with uploading my files. It might take days (or weeks), depending on how much data you want to upload and how fast your internet connection is. The unfortunate truth is that upload speeds are almost always much worse than download.
Let’s configure mail command. RPi will be able to email us about finished upload using a Gmail account.
sudo apt-get install ssmtp mailutils mpack
sudo nano /etc/ssmtp/ssmtp.conf
And set (or add if they are not there) these options
mailhub=smtp.gmail.com:587
hostname=raspberrypi
AuthUser=myraspberrypilogin@gmail.com
AuthPass=myraspberrypipassword
useSTARTTLS=YES
Now we should be able to send a test message
echo "email body" | mail -s "email subject" your@email.com
If everything worked fine, we can add email notification to our upload script
nano upload.sh
The whole script should look like this
find . -name "*.txt" | sort | while read file ; do
echo "Uploading $(basename "$file") to Amazon Glacier."
glacier-cmd upload --description "$(basename "$file")" photos "$file" && mv "$file" "uploaded"
done
echo "Selected files were uploaded successfully." | mail -s "Glacier uploads finished" your@email.com
Now, our script is going to email us when it’s finished.
9. Throttling upload
The last part of this tutorial is throttling upload speed, so that RPi doesn’t choke your internet connection
sudo apt-get install wondershaper
sudo wondershaper wlan0 100000 400
The limits that we’re setting are in kb, so the 400kb above equals 50kB. The first parameter is our network device, wlan0 for wifi, eth0 for wired connection. The second parameter is download speed, which you probably don’t want to limit. The third parameter is upload speed.
Run the following to clear limits
sudo wondershaper clear wlan0
You might want to add these commands to cron, so that RPi can use as much upload speed as possible and cut it back during the day. Here are the commands that you might add to cron to limit speed at 10 am and remove this limitation at 1am, every day
0 10 * * * root sudo wondershaper wlan0 100000 400
0 1 * * * root sudo wondershaper clear wlan0
That’s it, our RPi is ready to push tons of data to the cloud.
Other parts in this series
Posted 10 years ago
-