Tech Your Universe

Backups are something you must master to be a great system administrator.

You’e probably found this because you were looking for a simple backup solution. Yes, you’ve seen Amanda. And Bacula, but they aren’t simple. Amanda and Bacula are great products if you need all of their features — and if you are like me, I don’t want to spend time with my backups, I just want something that works.

My choice — rsnapshot. rsnapshot is a perl script that wraps around rsync. It’s most beautiful feature: it uses hard-links when it can, so if you are backing up the same file more than once, it just creates a link. This means backups only take up more space if the files change. I’ve heard that this is how Apple’s Time Machine works. I’m now using rsnapshot in multiple production environments. Here’s a quick how-to guide for how to set up reliable, robust, efficient, self-rotating backups in just a few minutes with rsnapshot.

1. Install rsnapshot, either by downloading/compiling the source code, or using this RPM for linux
2. Edit /etc/rsnapshot.conf for your settings. Warning: The config file makes a distinction between tabs and spaces. Make sure you use tabs! Pay special attention to these settings:
   1. snapshot_root — this is where your backups will be stored.
   2. The backup section will define which directories you want backed up.
      <br /> backup /etc/ localhost/<br /> backup /root/ localhost/<br /> backup /home/ localhost/<br />
   3. rsnapshot handles the rotation of backups for you! The interval section will define how many daily, weekly, etc. backups are kept. Example:
      <br /> interval hourly 6<br /> interval daily 7<br /> interval weekly 4<br /> interval monthly 3<br />
3. As a test, you can run: rsnapshot -v -t hourly and this will parse the config file, and show you the commands it will run when it runs hourly.
4. After you are done tweaking the config file, it’s time to add the crontab entries for the various backups. Scheduling is a bit tricky. Here’s mine as an example:
   <br /> # There is a pid file that will preven two from running at the same time.<br /> # This is why hourly starts after the others. Hourly should be skipped when daily/weekly/monthly is running.<br /> 19 */3 * * * nice rsnapshot -v hourly<br /> 18 1 * * * rsnapshot -v daily<br /> 17 2 * * 0 rsnapshot -v weekly<br /> 16 3 1 * * rsnapshot -v monthly<br />

That’s it! You’re ready to go. Your backups will be stored in rsnapshot_root.

Two words.

Thank.

God.

Ok, more than two words, because I know people are wondering how I feel.

First, I left Yahoo! because of Microsoft. Yes, I was wrong about the final outcome. When I left it looked like it was 80% likely that it was going to happen. I’m still correct in my assumpitions about what the hostile takeover attempt has done to the company. And I think I’ve found something better in my new job. I still feel I made the right decision.

Second, I am sure that Yahoo is better positioned to be successful alone, rather than with Microsoft in any way, and while this is definitely going to be a short term hit for shareholders, it is the better long term outcome for Yahoo to be independent. Read more about why the deal would have been a complete disaster.

Third, Steve Balmer, you can take this Yahoo/Google search deal and shove it up your a$$. You forced Yahoos hand, and now you are going to be a distant 3rd place, for a very long time. At least until Wikia Search takes it from you.

Finally, During this uncertain time at Yahoo, people have left. More today. Zawodney. JR Conlin. Both Yahoo icons. The disruption that this has had has put both Yahoo and Microsoft further away from reaching Google.

It’s now time for Yahoo to pick up the pieces and move on. Oh - and to spend another 12-24 months fighting shareholder lawsuits. Wouldn’t it be great if Yahoo countersued Microsoft for the damages?

More info at news.yahoo.com

Let’s say your laptop is stolen. Wouldn’t that be awful?

Now, what if you had a way to track down the person that took it and get it back?

Using Linux or Mac, it’s easy. let’s take a look at a script that will do this for you. It will take you less than 5 minutes to set up.

Save the above in /tmp/phone.bash (change $yourserver to a place where you can have a file hosted), then set add a crontab entry to have it run every 5 minutes:

*/5 * * * * /tmp/phone.bash

What does it do? Every 5 minutes, this script will run, and it will execute whatever code you have placed in the phonehome file on $yourserver. By default, I have my phonehome file just set to run true, which does nothing. But if my laptop gets stolen, I modify the phonehome file to include bash commands. I have the full power of bash on my laptop as soon as the thief connects to the internet. Imagine the possibilities. Here are some ideas:

1. traceroute - Let’s go ahead and get his IP address and where he is at. We should be able to give this to the police who can then contact his ISP and get his address. Save the output and copy it to $yourserver:
   

   traceroute > /tmp/traceroute.out
   scp /tmp/traceroute.out $yourserver:/tmp/theiftraceroute

   Tip: Set up ssh without a password to $yourserver so you can easily send information back and forth with rsync and/or scp.

2. Keystroke logger - Now let’s record everything he types. Using bash, we can download, install, and run a keystroke logger. Here’s one that works for OSX
3. Take a picture - If you have a Mac with a built in iSight camera, you can use isightcapture to take a pictures of the thief and send them to you!
   

   curl –silent "$urlforisightcapturescript" > /tmp/isightcapture
   chmod a+x /tmp/isightcapture
   /tmp/isightcapture –file /tmp/pictureoftheif$RANDOM
   scp /tmp/pictureoftheif* $yourserver:/tmp/

   Update:. See my post on automatically capturing pictures with isightcapture

Someone. Please. Steal my laptop. I can’t wait to use this.

Want to be an advanced debugger? My #1 Superman debugging tool is Linux’s strace. If you have ever run into problems where a user complains that the site is slow, and you can’t figure out why, you may want to give strace a try.

From http://sourceforge.net/projects/strace/:

strace is a system call tracer, i.e. a debugging tool which prints out a trace of all the system calls made by a another process/program.

In other words, strace tells you what a program is doing, at the C function call level. This is great for finding the problems where a page just "hangs" for no apparent reason. Let’s walk through what it takes to set up strace on Apache in a LAMP environment, with some real world examples that I’ve run into.

First, you’ll need to install strace, if it isn’t already installed. My favorite method is just yum install strace, but if you want to, you can download and compile it yourself.

Next, you will need a place where you can test the slow page. For the rest of this article, we will assume you have a development environment that is all to your own, where you can start/stop Apache at will, and no one else will be using it. Note: If a separate development environment isn’t available, I suggest running another Apache on a different port, say 81 instead of 80. This way you can still work on the production site without affecting end users.

Environment set up? Good. Let’s get down to debugging.

1. Start Apache in "Debug Mode" with the -X option. This has Apache start one process, instead of a bunch of children, and then all the requests will go through one process.
   

   httpd -X

2. In another terminal window, find the process id for the listening Apache that you just started. ps auxw | grep httpd should do the trick.
3. Once you have the process id, attach strace with the -p option:
   

   strace -p $processidofapacheprocess

4. Go to your browser and go to the url that is hanging. While it is running, watch the output from strace in your terminal window. You’ll see a ton of system calls stream by, but the important thing to look for is when it stops. What is it doing?

I’ve used this approach to find several "Superman" level problems (problems that other people spent at least a day trying to figure out what was going on — sometimes weeks). Here are some examples.

1. Sendmail hanging via PHP - The reported problem was that certain pages were slow (30-300 seconds). Load on the machines seemed fine, but certain requests were painfully slow. strace revealed that the PHP script was waiting for sendmail to come back with a response. Upon looking further, sendmail was doing a reverse dns lookup that was timing out, which resulted in a 30+ second delay. Problem resolved by reconfiguring sendmail.
2. PHP pages slow on an NFS server - The reported problem was a development environment with pages that were slow to load. strace revealed that the pages were hanging at a flock call to a directory that was mounted via NFS. Here’s the actual output from strace:
   

   …pages of output snipped…
   fcntl(24, F_SETFL, O_RDWR) = 0
   sendto(24, "incr toys:stats:request_with_ses"…, 40, MSG_DONTWAIT, NULL, 0) = 40
   poll([{fd=24, events=POLLIN|POLLERR|POLLHUP, revents=POLLIN}], 1, 500) = 1
   recvfrom(24, "76\r\n", 8192, MSG_DONTWAIT, NULL, NULL) = 4
   open("/home/phpsessions/sess_079113645a3da0fe50f68e4ce6ed58d2″, O_RDWR|O_CREAT, 0600) = 25
   flock(25, LOCK_EX

   So we can see here that the file /home/phpsessions/sess_079113645a3da0fe50f68e4ce6ed58d2 has been opened, and the flock call is hanging. Turns out NFS doesn’t deal well with flock. When we saw this, there was a big smack on the forehead. Why on earth were the sessions being stored via NFS anyway? Especially for a development server, where only one box needed to store it. To solve the problem, we changed the session.save_path in the php configuration file to a directory that was not on NFS.

3. Memcached hanging - Again, certain requests were hanging, causing pages to be slow to load. Again, strace to the rescue! Turns out PHP was hanging when talking to memcached. Once this was determined, we also ran strace on memcached, and found a bug with the particular memcached client we were using via PHP. We upgraded the memcached client to the latest version, and the problem was solved.

In all of the above cases, the problem could have been found through other means, but strace made it a much easier and faster to figure out where the slowdowns were.

There are other helpful uses of strace. In addition to finding hanging web pages, I’ve also used strace to find why/where Apache was segfaulting. Just run strace and look to see what the last thing it did. It should give you an indication of why the script stopped when it did.

Also, I’ve used Apache as a troubleshooting tool to find out where most of the time is being spent by analyzing the entire request.

Good luck in your adventures with strace, it’s been a big help for me. Feel free to leave a comment with your findings.