Tech Your Universe

Sure, you have a Gigabit card, but how can you check to see what speed you are running at in Linux? All your networking equipment must be Gigabit for it to work. There are two tools to do this, /usr/sbin/ethtool and /sbin/mii-tool

sudo /usr/sbin/ethtool eth0
Password:
Settings for eth0:
Supported ports: [ MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Current message level: 0×000000ff (255)
Link detected: yes

Enjoy.

 I took some time to improve upon someone else’s squid init script for Linux. I’m sharing it here:

    http://www.techyouruniverse.com/squid-init-script/

   Notable improvements: 

• Added clearcache method for clearing the squid cache. Surprisingly, this isn’t built into squid, and when you restart squid, you only clear the cache that is in memory. This clears the cache that is on disk too. Usage:
  

  /etc/rc.d/init.d/squid clearcache      

• Added configtest method for checking syntax of config file, just like the one for httpd.
• Optimized restart/stop for minimum downtime. Notably, the stop was set to allow http connections to die gracefully. I wasn’t happy with this because it caused 10 - 15 seconds of downtime whenever squid was restarted. Changed to use squid -k shutdown instead of squid -k interrupt, which closes the connections immediately instead of waiting for them to finish. Shut down takes 1-2 seconds now instead of 10-15.
• Cleaned up config variables
• Cleaned up formatting of output

This script has been used in production for a while, but if you find any issues, please leave a comment so that I can address them. Other comments/ suggestions appreciated. By the way, if you are the original author of the script, I’ll gladly give credit, leave a comment.

I am eager to see who will replace Google as the dominant search engine. It bothers me that one company has this much control of the Internet. And now, even dating:

http://www.google.com/romance/

Recently I’ve been with SEO for www.wikinvest.com, and I’ve been shocked at just how much power and control Google has over a business’ ability to succeed.

I think the Google revolt has begun. I already see it happening in the Bay Area technology conspiracy theorists (1, 2. People are getting fed up with Google’s faux-evilness.

Some people say ask.com is a likely candidate, but I think it will be a non-commercial (open source?) initiative. I’d like to see and take a community driven approach.

dmoz.org tried, but has lost credit because of slow updates to their directory. I submitted a legitimate site nearly 6 months ago, but it’s still not there as of 12/03/2007.

We need a search engine that is powered by the community, and non profit, and doesn’t suck. The ranking algorithm will be open source. Businesses will be presented objectively, not based on who pays more. The community will work together to fight spam. Users will contribute content indexing bandwidth and processing power with a distributed computing platform that utilizes people’s PCs, ala seti@home.

Google is fine for now. It does a good job and most people are happy, and it’s not bad enough to start a revolt. But as they continue down the evil path of complete control and monopoly — *cough* Microsoft — users will revolt. People will start to bolt. And when geeks get pissed, they find away around the problem — witness warez, napster, deCSS, and bittorent.

Update 1/2/2008: Holy shit. Wikipedia did it, they created a community based search. I’m going to call it now. This is the next Google Search.

Wikipedia Founder Brings Search Project

I’ve worked as a sys admin. I’ve managed system administrators. I’ve been a customer of a system admin. I’ve worked with great system administrators, and the not-so-great. Here are some thoughts on how to be the best there is.

1. Monitoring — This is the #1 secret. Monitoring is your automated Quality Assurance team. It will be there to save you, be there to point our your flaws, and enable you to fix problems before they are noticed by your customers. What’s worse than having something down for 24 hours? Having it down for two weeks before you even noticed. Monitor everything imaginable. In fact, monitor your monitors.
   • Service level monitoring — are the ports listening for every important service?
   • Smoketest monitoring — Regularly fetch a page just as an end user would, with a reasonable timeout. This ensures that all the pieces are working together and delivering real pages to end users.
   • Network monitoring — Is the network up? From how many different locations? Have at least one offsite network monitor.
   • Server stats — disk space, memory usage, cpu usage, network traffic, you get the idea. Ideally these are logged historically too, so you can pinpoint when the box started getting busy.

   How to administer all this monitoring? Nagios is the industry standard. It’s quite a learning curve and lots of set up, but well worth it.

2. Minimize manual processes — Let’s pretend you’re going on vacation for 3 months. What will break while you are gone? Will disks fill up because backups are rotated by hand? What process will stop working because it needs some sort of manual intervention? Put mechanisms in place so that these items are self managed. Your goal should be to automate yourself out of a job. You’ll never get there, but this type of thinking will improve the stability and robustness of your systems.
3. Be a Consistency Zealot - Do you have configuration files that are different for each server? Do you have one way of managing files for dev boxes and another for production? Staging? QA? Here’s a quick test that you can give yourself from bash to see how consistent you are. Try this:
   

   for box in $myboxes; do
   ssh $box md5sum /etc/my.cnf
   done

   How did you do? If you are like most systems, there were at least one file that was out of sync. Minimize the amount of these wherever practical, and aggressively defend this consistency. There must be a compelling reason to have things different. Tip: How about putting some monitors on critical configuration files so you know when they are changed?

4. Be a Simplicity Zealot — The number one rule of software also applies to the system administration world. Sure, you can build a hot failover mechanism so that you save the 5 minutes of time to switch manually, but are you sure the extra complexity is worth it? If your lead developer can’t come in and figure out what is going on and fix something in an emergency, you are probably doing something wrong. I have had a couple of people that didn’t understand my simplicity push at first, but after living and breathing it for a while, they now also swear by it. If in doubt, go with the simpler path.
5. Don’t be a dick — we’ve all worked with "that guy". He always talks about how stupid users are, and everyone is beneath them. He can’t even bother to stop and talk to you, because you obviously wouldn’t understand. He answers almost every question with either "It depends" or "Read the F^@$#ing manual". He enjoys pointing out how stupid your question was, and makes you ask it perfectly before he’ll give you an answer. Don’t be this guy. Instead, try to be the guy that everyone likes to be around.
6. Restores are worth more than gold — It’s critical to get backups right. People don’t want to have worry about backups, they just expect that they are there. With backups, you can either be a hero, or a severe disappointment. Run around without people knowing and back up everything they expect, plus some things they don’t. Try to make someone’s day at least twice a year by having something backed up that they need. Tip: Also make it easy to restore. It’s awful to have a backup of something, but the restore process takes longer than the usefulness of the backup.
7. Be Trustworthy. People have to trust you. You can read these people’s e-mails, and you often have access to their personal files. You are expected to answer your phone at all times. A solid working relationship requires a high degree of trust with all jobs, but it’s even more important as a system administrator. Hold your word as something to be cherished. Maintain high credibility by holding back your thoughts until you verify what you say. Make sure that everyone knows that they can trust you with their secrets, and be there to catch you with a much needed backup when they mess up.

These principals will guide you towards System Administration heaven, where users are smart, disks never fill up, network lights are always green, and developers can’t screw up your systems.

For anyone waiting to see how the early adopters do, I went to the Mac OSX Leopard Release at the Apple Store in San Francisco and bought myself a Family Pack. I’m happy to report that Leopard upgrade was successful on 3 different computers now.

• 1 Macbook Pro
• 1 Ibook G4
• 1 Imac G4

Very stable on all 3, and it seems faster to me. Maybe this is just the "fresh install" effect though.Important software that I can say works without any compatibility issues:

• Quicksilver
• Microsoft Office
• Cisco VPN
• Firefox
• Camino
• Adium
• smc fan control

My key "wow" features:

1. Spaces kicks ass. This was my biggest reason for upgrading.
2. Mail 3.0 support "todo list" that are stored on the IMAP server, so I can access them from anywhere.
3. Safari 3.0 has the coolest "find" feature I’ve ever seen in a browser.

Trac is decent bug tracking software that is getting a lot of attention these days. It’s very simple to use, which I like. It includes subversion integration so that you can walk through your source code tree and view diffs from one version to another. Handy.

However, Trac’s installation process has a lot of room for improvement in the simplicity department. Here are the steps I went through to install trac on Cent OS 5. I captured them so that others may find it easier to get Trac up and running. Enjoy.

If these instructions work for you, please leave a comment! If they don’t, please leave a comment!

1. Install python and its goodies.
   1. Use yum to install the base package: yum install python
   2. Install mod_python. This is so Apache can run the python scripts as a module:
      yum install mod_python
   3. Install MySQL-python so that python can interact with mysql
      1. Download and untar MySQL-python from http://sourceforge.net/projects/mysql-python
      2. cd $mysqlpythonsourcedir
      3. python setup.py build && python setup.py install
2. Install some devel packages that are needed for compiling trac and svn integration:
   yum install neon neon-devel python-devel swig
3. Install Clearsilver, a templating package that trac depends on.
   1. Download and untar clearsilver from http://www.clearsilver.net/downloads/
   2. Compile cd $clearsilversourcedir; ./configure && make && make install
4. Install trac!
   1. Download and untar trac from http://trac.edgewall.org/wiki/TracDownload
   2. cd $tracsourcedir; python ./setup.py install
   3. Initialize your first trac project:
      trac-admin $pathtoyourtracproject initenv
   4. Tell Apache about trac.. Add the following lines to your Apache configuration file:

        <Location />
          SetHandler mod_python
          PythonHandler trac.web.mod python_frontend
          PythonOption TracEnv $pathtoyourtracproject
          PythonOption TracUriRoot /trac/
        </Location>
      

Finally, restart httpd and try it out by going to:

http://$yourhost/trac/

If it doesn’t work, check your webserver error log for errors:
tail -f /var/log/httpd/error_log

Again, please leave a comment so that you may help others with the same problems you have.

Inevitably mysql replication gets borked. Some sql statement will fail, which stalls mysql replication. While this can be mitigated by restricting access on the slave, anyone who has used mysql replication in production knows that it can, and will happen. So plan for it.

If you see something like this when you run SHOW SLAVE STATUS Tip: Use the \G instead of the semicolon at the end of SQL statements, it produces more readable output.

You can try to skip the failed statement with:

mysql>STOP SLAVE();
mysql>SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1;
mysql>START SLAVE();

But that can’t solve all problems, as that sql statement you just skipped may have been needed to keep the data in sync. So what do you do? It’s time to reset the data on the slave. That’s right, wipe it and grab a fresh copy from the master. It’s the only way.

To grab a fresh copy of the data from the master, here you have several options.

1. Run a mysqldump on the master with –master-data and reimport on slave. Note that while the mysqldump is running on the master, tables are locked and the database is unusable
2. Stop mysql on the master and take a snapshot of all the data using tar or a filesystem snapshot
3. Use ibbackup. Note that ibbackup costs money and it will only work for InnoDB files, so you have to do something else for your MyISAM tables.

The problem with all of these solutions? They require downtime on the master database. That may be fine for some situations, but everywhere I’ve worked having downtime on the master data isn’t a good thing. So this is where you wish that you had thought ahead, and you had another slave that you could grab the master snapshot from. Aha!

Always have more than one slave. Put it on a piece of crap box some where. It’s another copy of your data for redundancy, and it gives you another place besides the master to grab the data from in the event that you need a snapshot of the data.

How do you rebuild a slave from another slave you ask? I’ll write up a step by step guide some day, but here’s a rough overview off the top of my head:

1. Stop mysql on the working slave
2. tar up all the files. This is everything in the "data" directory and the ibdata* files if you are using InnoDB.
3. Make a copy of the master.info file
4. Start mysql on the working slave
5. Stop mysql on the broken slave.
6. Untar all the data files.
7. Edit the my.cnf on the broken slave, and add "skip_slave_start=1″ in the [mysqld] section.
8. Start mysql on broken slave. Note that replication won’t be running yet because of the skip_slave_start line you added in the previous step.
9. Get the position from the master.info, and use "CHANGE MASTER TO" to reset the log file position on the broken slave to match the one on the working slave
10. Run START SLAVE on the working slave, check slave status with SHOW SLAVE STATUS to see if it worked.
11. Remove the skip_slave_start from the my.cnf file so the slave will start next time mysql is restarted.

YAML is a syntax for configuration files.

Huh?

Why on earth do we need another technology for config files?

I first ran across YAML while working with Symfony. YAML’s use in PHP is especially troubling, because PHP has a built in function for parsing config files, parse_ini_file(). This config file syntax is the same as the php.ini file, so it is well known by all PHP system administrators. It’s human readable, supports basic name/value pairs, and allows for comments.

It’s also high performance, since it’s a built in function, config files are parsed with the speed of C instead of text processing with PHP. This may not matter much for Joe Bloe’s blog website, but when you have lots of users, this makes a difference.

From what I saw with it’s use within symfony, everything could have been easily done using parse_ini_file(), and to make matters worse, when it was noticed that there where performance problems with parsing YAML, the Symfony authors decided to add a caching layer. Great. More complexity. This violates the number one rule of good software: Simplicity.

The Symfony authors should have used an existing technology that was built into PHP for handling config settings. Don’t make the same mistake.

If only our wives behaved like our servers:

For those that don’t get it, sudo is a command in unix that executes the command as the super user, root. Often times you are working with unix and you try to do something as a normal user, but get a permission denied message, and you use sudo with the same command to run it as root, and it works.

-Nick

Very interesting write up on what could be coming.

http://sethgodin.typepad.com/seths_blog/2007/01/web4.html

Some of us are likely to freak out about privacy concerns… but I for one am eager to embrace some of the cool technology innovations mentioned here, such as:

"I’m late for a dinner. My GPS phone knows this (because it has my calendar, my location, and the traffic status). So, it tells me, and then it alerts the people who are waiting for me."

-Nick