Archive for the 'technology' Category

Published by nick on 20 Jun 2008

Howto - Simple backups for Linux using rsnapshot

Backups are something you must master to be a great system administrator.

You’e probably found this because you were looking for a simple backup solution. Yes, you’ve seen Amanda. And Bacula, but they aren’t simple. Amanda and Bacula are great products if you need all of their features — and if you are like me, I don’t want to spend time with my backups, I just want something that works.

My choice — rsnapshot. rsnapshot is a perl script that wraps around rsync. It’s most beautiful feature: it uses hard-links when it can, so if you are backing up the same file more than once, it just creates a link. This means backups only take up more space if the files change. I’ve heard that this is how Apple’s Time Machine works. I’m now using rsnapshot in multiple production environments. Here’s a quick how-to guide for how to set up reliable, robust, efficient, self-rotating backups in just a few minutes with rsnapshot.

  1. Install rsnapshot, either by downloading/compiling the source code, or using this RPM for linux
  2. Edit /etc/rsnapshot.conf for your settings. Warning: The config file makes a distinction between tabs and spaces. Make sure you use tabs! Pay special attention to these settings:
    1. snapshot_root — this is where your backups will be stored.
    2. The backup section will define which directories you want backed up.
      <br /> backup /etc/ localhost/<br /> backup /root/ localhost/<br /> backup /home/nick/ localhost/<br />
    3. rsnapshot handles the rotation of backups for you! The interval section will define how many daily, weekly, etc. backups are kept. Example:
      <br /> interval hourly 6<br /> interval daily 7<br /> interval weekly 4<br /> interval monthly 3<br />
  3. As a test, you can run: rsnapshot -v -t hourly and this will parse the config file, and show you the commands it will run when it runs hourly.
  4. After you are done tweaking the config file, it’s time to add the crontab entries for the various backups. Scheduling is a bit tricky. Here’s mine as an example:
    <br /> # There is a pid file that will preven two from running at the same time.<br /> # This is why hourly starts after the others. Hourly should be skipped when daily/weekly/monthly is running.<br /> 19 */3 * * * nice rsnapshot -v hourly<br /> 18 1 * * * rsnapshot -v daily<br /> 17 2 * * 0 rsnapshot -v weekly<br /> 16 3 1 * * rsnapshot -v monthly<br />

That’s it! You’re ready to go. Your backups will be stored in rsnapshot_root.

Published by nick on 05 Jun 2008

Phone Home Script to Protect Your Laptop

Let’s say your laptop is stolen. Wouldn’t that be awful?

Now, what if you had a way to track down the person that took it and get it back?

Using Linux or Mac, it’s easy. let’s take a look at a script that will do this for you. It will take you less than 5 minutes to set up.

#!/bin/bash

curl –silent “http://$yourserver/phonehome?homelaptop” > /tmp/phonehome
if [ -s /tmp/phonehome ]; then
sh /tmp/phonehome
fi

Save the above in /tmp/phone.bash (change $yourserver to a place where you can have a file hosted), then set add a crontab entry to have it run every 5 minutes:

*/5 * * * * /tmp/phone.bash

What does it do? Every 5 minutes, this script will run, and it will execute whatever code you have placed in the phonehome file on $yourserver. By default, I have my phonehome file just set to run true, which does nothing. But if my laptop gets stolen, I modify the phonehome file to include bash commands. I have the full power of bash on my laptop as soon as the thief connects to the internet. Imagine the possibilities. Here are some ideas:

  1. traceroute - Let’s go ahead and get his IP address and where he is at. We should be able to give this to the police who can then contact his ISP and get his address. Save the output and copy it to $yourserver:

    traceroute > /tmp/traceroute.out
    scp /tmp/traceroute.out $yourserver:/tmp/theiftraceroute

    Tip: Set up ssh without a password to $yourserver so you can easily send information back and forth with rsync and/or scp.

  2. Keystroke logger - Now let’s record everything he types. Using bash, we can download, install, and run a keystroke logger. Here’s one that works for OSX
  3. Take a picture - If you have a Mac with a built in iSight camera, you can use isightcapture to take a pictures of the thief and send them to you!

    curl –silent “$urlforisightcapturescript” > /tmp/isightcapture
    chmod a+x /tmp/isightcapture
    /tmp/isightcapture –file /tmp/pictureoftheif$RANDOM
    scp /tmp/pictureoftheif* $yourserver:/tmp/

Someone. Please. Steal my laptop. I can’t wait to use this.

Published by nick on 02 Jun 2008

Debugging web apps with strace

Want to be an advanced debugger? My #1 Superman debugging tool is Linux’s strace. If you have ever run into problems where a user complains that the site is slow, and you can’t figure out why, you may want to give strace a try.

From http://sourceforge.net/projects/strace/:

strace is a system call tracer, i.e. a debugging tool which prints out a trace of all the system calls made by a another process/program.

In other words, strace tells you what a program is doing, at the C function call level. This is great for finding the problems where a page just “hangs” for no apparent reason. Let’s walk through what it takes to set up strace on Apache in a LAMP environment, with some real world examples that I’ve run into.

First, you’ll need to install strace, if it isn’t already installed. My favorite method is just yum install strace, but if you want to, you can download and compile it yourself.

Next, you will need a place where you can test the slow page. For the rest of this article, we will assume you have a development environment that is all to your own, where you can start/stop Apache at will, and no one else will be using it. Note: If a separate development environment isn’t available, I suggest running another Apache on a different port, say 81 instead of 80. This way you can still work on the production site without affecting end users.

Environment set up? Good. Let’s get down to debugging.

  1. Start Apache in “Debug Mode” with the -X option. This has Apache start one process, instead of a bunch of children, and then all the requests will go through one process.

    httpd -X

  2. In another terminal window, find the process id for the listening Apache that you just started. ps auxw | grep httpd should do the trick.
  3. Once you have the process id, attach strace with the -p option:

    strace -p $processidofapacheprocess

  4. Go to your browser and go to the url that is hanging. While it is running, watch the output from strace in your terminal window. You’ll see a ton of system calls stream by, but the important thing to look for is when it stops. What is it doing?

I’ve used this approach to find several “Superman” level problems (problems that other people spent at least a day trying to figure out what was going on — sometimes weeks). Here are some examples.

  1. Sendmail hanging via PHP - The reported problem was that certain pages were slow (30-300 seconds). Load on the machines seemed fine, but certain requests were painfully slow. strace revealed that the PHP script was waiting for sendmail to come back with a response. Upon looking further, sendmail was doing a reverse dns lookup that was timing out, which resulted in a 30+ second delay. Problem resolved by reconfiguring sendmail.
  2. PHP pages slow on an NFS server - The reported problem was a development environment with pages that were slow to load. strace revealed that the pages were hanging at a flock call to a directory that was mounted via NFS. Here’s the actual output from strace:

    …pages of output snipped…
    fcntl(24, F_SETFL, O_RDWR) = 0
    sendto(24, “incr toys:stats:request_with_ses”…, 40, MSG_DONTWAIT, NULL, 0) = 40
    poll([{fd=24, events=POLLIN|POLLERR|POLLHUP, revents=POLLIN}], 1, 500) = 1
    recvfrom(24, “76\r\n”, 8192, MSG_DONTWAIT, NULL, NULL) = 4
    open(”/home/phpsessions/sess_079113645a3da0fe50f68e4ce6ed58d2″, O_RDWR|O_CREAT, 0600) = 25
    flock(25, LOCK_EX

    So we can see here that the file /home/phpsessions/sess_079113645a3da0fe50f68e4ce6ed58d2 has been opened, and the flock call is hanging. Turns out NFS doesn’t deal well with flock. When we saw this, there was a big smack on the forehead. Why on earth were the sessions being stored via NFS anyway? Especially for a development server, where only one box needed to store it. To solve the problem, we changed the session.save_path in the php configuration file to a directory that was not on NFS.

  3. Memcached hanging - Again, certain requests were hanging, causing pages to be slow to load. Again, strace to the rescue! Turns out PHP was hanging when talking to memcached. Once this was determined, we also ran strace on memcached, and found a bug with the particular memcached client we were using via PHP. We upgraded the memcached client to the latest version, and the problem was solved.

In all of the above cases, the problem could have been found through other means, but strace made it a much easier and faster to figure out where the slowdowns were.

There are other helpful uses of strace. In addition to finding hanging web pages, I’ve also used strace to find why/where Apache was segfaulting. Just run strace and look to see what the last thing it did. It should give you an indication of why the script stopped when it did.

Also, I’ve used Apache as a troubleshooting tool to find out where most of the time is being spent by analyzing the entire request.

Good luck in your adventures with strace, it’s been a big help for me. Feel free to leave a comment with your findings.

Published by nick on 25 Apr 2008

Debugging rules

My co-worker, Artur Bergman, gave a talk today at the Web 2.0 Expo.

He highlighted these rules of debugging, and they were great, so I wanted to share.

  1. Understand the system
  2. Make it fail
  3. Quit thinking and look
  4. Divide and conquer
  5. Change one thing at a time
  6. Keep an audit trail
  7. Check the plug
  8. Get a fresh view
  9. If you didn’t fix it, it ain’t fixed.

Brilliant! See http://www.debuggingrules.com/

Published by nick on 11 Mar 2008

Wikinvest wins award for Best Business website of 2007

In addition to being full time at Yahoo!, another company I work for part time, Wikinvest.com, won an award at the 11th Annual South by Southwest Interactive Web Awards for best website of 2007 in the business category. See the complete list of winners

Hooray!

Shameless Plug:
Wikinvest is a website that applies the concept of wikipedia (community edited content) to the world of investing. We’ve got some great content, some very interesting expansions to the MediaWiki software, and the Search is pretty damned cool do (I built that). ;-)

If you are into investing, I encourage you to take a look

Published by nick on 07 Mar 2008

Is Microsoft changing it’s evil ways?

As a Yahoo! employee that stands emphatically against Microsoft’s hostile takeover, I’ve been pretty vocal about how awful Microsoft is because of their poor technology, their anti-open source stance, and their anti competitive history. So much so that Yahoo! management has asked me to “tone it down”. Ha. Ever since grade school, I do what I think is right, not what I am told to do.

Could I be wrong? Could Microsoft be changing it’s ways?

First, Microsoft announced that they were open-sourcing some of their platforms, a clear attempt at making-good with the open source community. I cautiously applauded this when it happened, even though I think that the timing was very convenient for appeasing Yahoos.

Today I ran across an article by Robert Scoble, where he highlights what good Microsoft has done in the last 6 months, and I agree the results are encouraging.

I just had dinner with a bunch of Italy’s top tech bloggers and technologists and Marc Canter. Plus I’ve been talking with people all day long. Microsoft hit major Internet home runs today with its announcements, based on what I’m hearing from formerly-skeptical developers.I haven’t heard this level of excitement about Microsoft’s Internet Strategy in years.

Interesting story, it’s worth a read. I support a change of heart by Microsoft, and I hope it’s genuine.

Not believing that Microsoft could really be trying to be a good citizen, I dug deeper and found an article by Dana Gardner, and he poses some interesting questions on what Microsoft’s ulterior motive may be:

And that raises the same old questions. Will the power increase to a point where the openness declines? Will the standards over time be increasingly set by the de facto marker leader? Will the Internet and its efficiencies work best for consumers and users, or those that can manipulate it best?

See the full story hereHmm. What are they up to? We will see. Most of the past 10 years of Microsoft’s business practices are marred with bad karma. If they are good citizens for the next 5, they can show the world that they have changed.

Published by nick on 28 Feb 2008

Microsoft’s anti-competitive history

My recent blog posts on Microsoft’s Hostile Takeover of Yahoo have painted me, to some, as an open-source zealot, which maybe I am. However, there is another reason why Yahoos won’t work for Microsoft. It’s because of Microsoft’s history of anti-competitive business practices.

I wrote this as a one-liner before, but I decided to expand on this a bit. First, let’s look at the definition of competition. From dictionary.com:

1. of, pertaining to, involving, or decided by competition: competitive sports; a competitive examination.
2. well suited for competition; having a feature that makes for successful competition: a competitive price.
3. having a strong desire to compete or to succeed.
4. useful to a competitor; giving a competitor an advantage: He was careful not to divulge competitive information about his invention.

In business, competition drives success, which — ultimately — benefits consumers. The Bureau of Competition seems to think that American companies should avoid monopolistic practices. Karl Marx didn’t agree. And there are plenty of others that would say that government shouldn’t be involved in such matters.

Through all of these different stances on competition — they all agree that fair play is a must, or consumers suffer. Ok, check, most people would agree that healthy competition amongst business is good for consumers. So now what are some anti-competitive practices? Wikipedia serves us pretty well here: http://en.wikipedia.org/wiki/Anti-competitive_practices

From this page, I took a list of practices, and then we will see if Microsoft has done with each of these.

Action Description Microsoft?
Dumping Where products are sold into a market at a low price which renders competition impossible, in order to wipe out competitors. No examples found
Exclusive dealing Where a retailer or wholesaler is tied to purchase from a supplier. Yes. Microsoft works out exclusive deals with hardware manufacturers to lock them into selling Windows, and only Windows, on desktop machines.
Barriers to entry (to an industry) designed to avoid the competition that new entrants would bring. Yes. Entering the PC Desktop market is very difficult because Microsoft Windows has interoperability challenges with other Operating Systems. Some consumers stick with Microsoft Windows because they feel they have no choice if they want to use all of their programs, and this market share dominance prevents competitors from easily introducing new, competing operating systems that would benefit consumers
Price fixing Where companies collude to set prices, effectively dismantling the free market. Yes. The European Union seems to think so, after just fining Microsoft 899 million Euro for its practices
Refusal to deal E.g., two companies agree not to use a certain vendor Yes. According to this paper, the European and Microsoft anti trust cases cited “refusal to deal”
Dividing territories e.g., you get everything west of the Mississippi, we take everything east No examples found - not really applicable to software
Limit Pricing Where the price is set by a monopolist to discourage economic entry into a market. Yes. Windows is bundled with hardware at reduced pricing, sacrificing short term gains for longer term market share and profit from other software such as Microsoft Office, raising the barrier of entry for competition.
Product tying Where products that aren’t naturally related must be bought together; this prevents consumer choice. Yes. Two words. Internet Explorer. See full details on the Microsoft Anti-trust case in the U.S. and EU
Resale price maintenance Where resellers are not allowed to set prices independently. Yes. To be fair, this is common practice
Coercive monopoly All potential competition is barred from entering the market Yes. Reference above or below examples.

In looking at how many anti-competitive business practices Microsoft employs, we can see why Microsoft is a quintessential example of an anti-competitive company. Still not convinced? Let’s look at some specific notable examples in Microsoft’s history.

Case Study #1 - Microsoft vs. Apple, through anti-competitive business practices, Microsoft dominates it’s way to large market share to become the largest software company, even though Apple had a better product.

Case Study #2 - Microsoft vs. Sun. In an attempt to kill Java, Microsoft stops bundling Java with Windows

Case Study #3 - Microsoft vs. Netscape/AOL. Microsoft employes anti-competitive business practices to take over the browser market, going as far as intentionally crippling Netscape in releases of Internet Explorer. The United States Government steps in and penalizes Microsoft.

Case Study #4 - Microsoft vs. Linux via SCO. SCO receives investment help from Microsoft, and suddenly SCO sues effectively the entire Linux community with the new found funding, spreading FUD everywhere for businesses running Linux. Interesting coincidence.

Case Study #5 - Microsoft vs Real Networks. They settled a suit in 2005

Case Study #6 - Microsoft vs Novell. They settled a suit in 2004

Microsoft has an insatiable appetite to win, at any cost. They play dirty, and seem to disregard the moral implications. They’ll trip you in a foot race, foul you in basketball, or bite your ear a boxing match. All in the name of winning. Now the key question: When does the desire to win become unhealthy?
My answer: When it compromises your integrity.

I have four kids, and although I’m not perfect, I try very hard to lead by example. Among other things, I teach honesty and integrity as a core principal. My kids know that integrity is more important than winning. In my opinion, Microsoft has shown that winning is more important than integrity.

Having integrity as a core value is a critical factor in me choosing what kind of company I want to work for. How then, can I have a position where morality and integrity are so important to me, and work for Microsoft?

The answer: I can’t. And I won’t.

Update on March 28th: I’ve decided to leave Yahoo! because of the impending Microsoft takeover.

Published by nick on 26 Feb 2008

Bay area recruiters salivating over Yahoo talent

With a certain percentage of Yahoos unwilling to work for Microsoft, recruiters are in a feeding frenzy. They know that this is a great opportunity to snatch up employees that would have otherwise been perfectly content with Yahoo! for many years to come.

Break off into a quick tangent to dispel a myth — Some may say, what about the Change in Control Severance Plan that was put in place to “line the pockets of Yahoo! employees”? Well, in my opinion, it does not benefit most Yahoo employees as the media would have us believe. Some may think that this is incentive for Yahoos to stay. However, it only kicks in if Microsoft fires Yahoo employee, or the Yahoo employee leaves for “Good Reason”, ie, if they ask Yahoo employee to relocate to Redmond. Does “Good Reason” include the moral conflict of working for a company with a history of unscrupulous business practices? Apparently not, but that’s another story that I won’t discuss publicly.

</tangent>

Anyway, back to Pavlovian recruiters. In a less-than-scientific-completely-informal survey, a co-worker in a nearby cube says he has received 8 phone calls and 2 invitations from LinkedIn in the last two weeks. I just counted 19 e-mails, phone calls, or LinkedIn invitations I’ve received in the past two weeks. Search engine traffic to my resume has gone up 300% in the past month. In looking at the keywords that users are searching for to find my resume in the last few days, there are some interesting findings. Note how much “yahoo” is being searched on:

nick sullivan yahoo
software architect sample resumes
“my resume” and (nyc or “new york”) and (”java programmer” or “software programmer” or “database design”)
“nick sullivan” resume
technical yahoo” php perl mysql
(intitle:”resume for” | intitle:”resume of”) (programmer or programer or developer) framework
architect resume
chief software architect resume
css ajax web content develop (inurl:resume | intitle:resume) 94000..95000
database and engineer and architect and php (intitle:resume) 94002..95196
database engineer architect php (intitle:resume) 94002..95196
evox sunnyvale
intitle:resume inurl:resume network engineer “load balancing” “new york”
lead software architect resume
nick sullivan resume
php resume santa clara
project manager resume, automotive
resume builder software architect
resume yahoo maps development lead
resumes+php developer+ct
sample cv of software architect
sample resume of a chief software architect
technical architect and resume and new york
yahoo php developer resume
yahoo sunnyvale resume

Now, the deeper question. What does this mean for Yahoo!? What does this mean for Microsoft? This can’t be good for Yahoo, and it’s not good for Microsoft either, even if the hostile takeover fails. Why? Let’s take a look.

  • Yahoos are going to to Google — Where are the Yahoos that leave disgruntled most likely to go? Yes, that’s right, they will go to the very competitor that Microsoft is supposedly teaming up with Yahoo to beat! Great job Microsoft, you are kicking the best talent from your #2 competitor to your #1 competitor. Yet another argument that debunks the team up against Google myth.
  • Yahoos are going to startups that will compete with Microsoft — Freed from the burdens of getting work done in a large company (read: it’s ok to hack things together at a startup), innovation will explode. And when they get there, will they call up and recruit their former Yahoo! co-workers? You bet they will!
  • Yahoos are forming start ups of their own that will compete with Microsoft — I’ve been approached by 2 different Yahoo co-workers that are starting companies of their own, ideas that have been lingering for a while but were jump started because of the Microsoft threat. One is an online payment system (I can’t say more), and the other is a meta search for travel. These 2 people were otherwise content Yahoo employees that would have gladly continued their employment.

The climate in the Bay Area seems more frenzied than I’ve seen since 1999. With new startups being formed and existing startups being flooded with good talent, we are about to see a surge in innovation. We should see some very interesting products and ideas hit the market in 3-6 months.

Published by nick on 22 Feb 2008

Obama or Clinton - please save Yahoo

BusinessWeek reports that there is a surge in mergers and acquisitions right now because of the pending election. Interesting.

In particular, Obama reportedly plans to take a strong stance on antitrust issues:

Obama criticized the Bush Administration for “the weakest record of antitrust enforcement of any administration in the last half century.” He pledged to “reinvigorate” enforcement, and cited health care, including the insurance and pharmaceutical industries, as a sector where the lack of competition has raised prices for consumers.

Hmm. I’m starting to like this guy, even though I don’t think he will win. According to the article, democrats are more supportive of anti-trust legislation than republicans, which is consistent with the two parties’ respective views.

Jerry Yang, here is a new tactic for avoiding a Microsoft hostile takeover: Stall until we have a Democratic President. :)

See the Full Story

Published by nick on 21 Feb 2008

Microsoft to Yahoos: Look - we are open!

Microsoft today announced that they were going to be “opening up” their platform.

Microsoft today announced a set of broad-reaching changes to its technology and business practices to increase the openness of its products and drive greater interoperability, opportunity and choice. These changes are codified into four new interoperability principles and corresponding actions: 1) ensuring open connections; 2) promoting data portability; 3) enhancing support for industry standards; and 4) fostering more open engagement with customers and the industry, including open source communities.

More info here.

My take?

This is a good step in the right direction, even if it’s just talk. It helps address the “Key reason #1″ on the list of reasons why Yahoos won’t work for Microsoft.

To that end, very strategic timing. Could they be trying to make Yahoo employees feel better about their historical anti-open source stance?

With this news Microsoft gets 10 goodness points.

Next »