Tech Your Universe

Sure, you have a Gigabit card, but how can you check to see what speed you are running at in Linux? All your networking equipment must be Gigabit for it to work. There are two tools to do this, /usr/sbin/ethtool and /sbin/mii-tool

sudo /usr/sbin/ethtool eth0
Password:
Settings for eth0:
Supported ports: [ MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Current message level: 0×000000ff (255)
Link detected: yes

Enjoy.

 I took some time to improve upon someone else’s squid init script for Linux. I’m sharing it here:

    http://www.techyouruniverse.com/squid-init-script/

   Notable improvements: 

• Added clearcache method for clearing the squid cache. Surprisingly, this isn’t built into squid, and when you restart squid, you only clear the cache that is in memory. This clears the cache that is on disk too. Usage:
  

  /etc/rc.d/init.d/squid clearcache      

• Added configtest method for checking syntax of config file, just like the one for httpd.
• Optimized restart/stop for minimum downtime. Notably, the stop was set to allow http connections to die gracefully. I wasn’t happy with this because it caused 10 - 15 seconds of downtime whenever squid was restarted. Changed to use squid -k shutdown instead of squid -k interrupt, which closes the connections immediately instead of waiting for them to finish. Shut down takes 1-2 seconds now instead of 10-15.
• Cleaned up config variables
• Cleaned up formatting of output

This script has been used in production for a while, but if you find any issues, please leave a comment so that I can address them. Other comments/ suggestions appreciated. By the way, if you are the original author of the script, I’ll gladly give credit, leave a comment.

For anyone waiting to see how the early adopters do, I went to the Mac OSX Leopard Release at the Apple Store in San Francisco and bought myself a Family Pack. I’m happy to report that Leopard upgrade was successful on 3 different computers now.

• 1 Macbook Pro
• 1 Ibook G4
• 1 Imac G4

Very stable on all 3, and it seems faster to me. Maybe this is just the "fresh install" effect though.Important software that I can say works without any compatibility issues:

• Quicksilver
• Microsoft Office
• Cisco VPN
• Firefox
• Camino
• Adium
• smc fan control

My key "wow" features:

1. Spaces kicks ass. This was my biggest reason for upgrading.
2. Mail 3.0 support "todo list" that are stored on the IMAP server, so I can access them from anywhere.
3. Safari 3.0 has the coolest "find" feature I’ve ever seen in a browser.

Trac is decent bug tracking software that is getting a lot of attention these days. It’s very simple to use, which I like. It includes subversion integration so that you can walk through your source code tree and view diffs from one version to another. Handy.

However, Trac’s installation process has a lot of room for improvement in the simplicity department. Here are the steps I went through to install trac on Cent OS 5. I captured them so that others may find it easier to get Trac up and running. Enjoy.

If these instructions work for you, please leave a comment! If they don’t, please leave a comment!

1. Install python and its goodies.
   1. Use yum to install the base package: yum install python
   2. Install mod_python. This is so Apache can run the python scripts as a module:
      yum install mod_python
   3. Install MySQL-python so that python can interact with mysql
      1. Download and untar MySQL-python from http://sourceforge.net/projects/mysql-python
      2. cd $mysqlpythonsourcedir
      3. python setup.py build && python setup.py install
2. Install some devel packages that are needed for compiling trac and svn integration:
   yum install neon neon-devel python-devel swig
3. Install Clearsilver, a templating package that trac depends on.
   1. Download and untar clearsilver from http://www.clearsilver.net/downloads/
   2. Compile cd $clearsilversourcedir; ./configure && make && make install
4. Install trac!
   1. Download and untar trac from http://trac.edgewall.org/wiki/TracDownload
   2. cd $tracsourcedir; python ./setup.py install
   3. Initialize your first trac project:
      trac-admin $pathtoyourtracproject initenv
   4. Tell Apache about trac.. Add the following lines to your Apache configuration file:

        <Location />
          SetHandler mod_python
          PythonHandler trac.web.mod python_frontend
          PythonOption TracEnv $pathtoyourtracproject
          PythonOption TracUriRoot /trac/
        </Location>
      

Finally, restart httpd and try it out by going to:

http://$yourhost/trac/

If it doesn’t work, check your webserver error log for errors:
tail -f /var/log/httpd/error_log

Again, please leave a comment so that you may help others with the same problems you have.

Inevitably mysql replication gets borked. Some sql statement will fail, which stalls mysql replication. While this can be mitigated by restricting access on the slave, anyone who has used mysql replication in production knows that it can, and will happen. So plan for it.

If you see something like this when you run SHOW SLAVE STATUS Tip: Use the \G instead of the semicolon at the end of SQL statements, it produces more readable output.

You can try to skip the failed statement with:

mysql>STOP SLAVE();
mysql>SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1;
mysql>START SLAVE();

But that can’t solve all problems, as that sql statement you just skipped may have been needed to keep the data in sync. So what do you do? It’s time to reset the data on the slave. That’s right, wipe it and grab a fresh copy from the master. It’s the only way.

To grab a fresh copy of the data from the master, here you have several options.

1. Run a mysqldump on the master with –master-data and reimport on slave. Note that while the mysqldump is running on the master, tables are locked and the database is unusable
2. Stop mysql on the master and take a snapshot of all the data using tar or a filesystem snapshot
3. Use ibbackup. Note that ibbackup costs money and it will only work for InnoDB files, so you have to do something else for your MyISAM tables.

The problem with all of these solutions? They require downtime on the master database. That may be fine for some situations, but everywhere I’ve worked having downtime on the master data isn’t a good thing. So this is where you wish that you had thought ahead, and you had another slave that you could grab the master snapshot from. Aha!

Always have more than one slave. Put it on a piece of crap box some where. It’s another copy of your data for redundancy, and it gives you another place besides the master to grab the data from in the event that you need a snapshot of the data.

How do you rebuild a slave from another slave you ask? I’ll write up a step by step guide some day, but here’s a rough overview off the top of my head:

1. Stop mysql on the working slave
2. tar up all the files. This is everything in the "data" directory and the ibdata* files if you are using InnoDB.
3. Make a copy of the master.info file
4. Start mysql on the working slave
5. Stop mysql on the broken slave.
6. Untar all the data files.
7. Edit the my.cnf on the broken slave, and add "skip_slave_start=1″ in the [mysqld] section.
8. Start mysql on broken slave. Note that replication won’t be running yet because of the skip_slave_start line you added in the previous step.
9. Get the position from the master.info, and use "CHANGE MASTER TO" to reset the log file position on the broken slave to match the one on the working slave
10. Run START SLAVE on the working slave, check slave status with SHOW SLAVE STATUS to see if it worked.
11. Remove the skip_slave_start from the my.cnf file so the slave will start next time mysql is restarted.

Who is this Jason guy and what does he want with our data? JSON stands for JavaScript Object Notation, and I think there are some pretty compelling reasons to use it all the time instead of php’s serialize() function, and maybe even to replace XML under some almost all circumstances.

Why use JSON instead of XML for Asynchronous Javascript requests for your favourite web application? Well, why not? After all, it’s the simplest approach.

We’re going to build a widget for looking up the cities within 5 miles of a user supplied zip code. We want the user to enter their zip code into a form, and without reloading the page, use Javascript to go get data from our web service, and display it on the page. Let’s look at using JSON for the data type verses using XML.

JSON for web service

Server side:


$locations=getLocationsForZip($zip);
echo json_encode ($locations);


Client side:


var json=fetch(url);
var locations=eval(json);


XML approach

Server side:

$locations=getLocationsForZip($zip);
if (!empty($locations)){
  echo '<?xml version="1.0" encoding="utf-8"?>' . "\n";
  echo "<locations>\n";
  foreach ($locations as $location){
    echo "<location>\n";
    echo "<city>" . htmlspecialchars(utf8_encode($city)) . "</city>\n";
    echo "<state>" . htmlspecialchars(utf8_encode($state)) . "</state>\n";
    echo "<country>" . htmlspecialchars(utf8_encode($country)) . "</country>\n";
   echo "</location>\n";
  }
  echo "</locations>\n";
}

Client side:


var xml=fetch(url);
// TODO: write nasty code for parsing XML into DOM
// More nasty code to iterate through the DOM object it to put it into a usable array.


Now, that makes the most sense for Asynchronous Javascript requests, but going further, does it make sense to do it even for normal data transport mechanisms?

Language Neutral Data Storage

We need to store data on the file system or in a database that needs to be programming language independent. Historically, XML has been the obvious choice for this task. I’ve seen some people use serialized PHP (yuck). You could roll your own pipe delimited or CSV kludge. I think JSON is a best choice.

1. JSON is higher performance than XML, both in construction and parsing. Those of us who have [tried] to build scalable applications using XML/XSL have learned… not to.
2. JSON is language neutral, and built in! Every major language now has a json encode/decode capabilities.
   • PHP
   • Perl
   • C
   • Java)
   • Many more on Google
3. JSON is simpler. Just run json_encode() via PHP, and then ‘eval’ in javascript, and you’re done. See above examples.
4. JSON is more compact than XML. Less data on the disk, less data over the wire.
5. JSON contains character set information, and handles encoding issues for you (for the most part) — PHP’s serialize does NOT–, and this will cause problems for you when you internationalize
6. JSON maintains structure and objects, unlike pipe delimited or CSV hacks.

The more I use JSON for building my web applications, the more I’m finding that my reasons for using XML are fading away. Fast. XML is more difficult to deal with.

If you really feel compelled to use XML, do the rest of us a favor and make your webservice support both formats. Tip: Build a RESTful webservice and allow for a .json extension in your url in addition to .xml.

Props to Chris Cowan for helping me to see the light, and Douglas Crockford for evangelizing the use of Javascript.

-Nick

The most common reason that people cite for using a database abstraction layer is — "You can change the underlying database and your application doesn’t need to be rewritten."

From an academic perspective, this makes sense, and it’s hard to argue with. Especially in an interview or when you are talking to the fresh college grad with a Computer Science degree.

In the real world, it doesn’t happen.

Show me one person that has built their application with one database in mind, and then switched to another without having to rewrite some code, and I’ll show you 200 others that built their project using a database abstraction layer and they will never change the underlying database.

The concept of database neutrality is valuable under certain circumstances. If you are writing an open source software platform such as MediaWiki, and people that you don’t know will be installing the software on their own servers, and you want them to be able to use the database of their choice, then using a database abstraction layer for database neutrality is a clear winner.

There are other good reasons to use database abstraction layers:

• Convenience methods that you want to be able to reuse. My personal favorite example is getRowFromSql(), where you give it a sql statement that returns 1 row, and it will return an associative array with the columns/values.
• Common configuration settings. Ie, your username, password, and host information are all in one place.
• Central configuration of write vs. read connections. Do this right from the start, because you’ll need it when you go to scale mysql.
• Reuse of database connections for performance

These are enough for me, and they are the reason I use database abstraction layers. But if the only reason you are using a database abstraction layer is so that you can easily switch your database to something else in the future, you’re nuts, and you may want to make sure you are following the number one rule of software.

Me personally, I just have a simple class that extends PDO.

Rule #1 for building software. Simplicity.

Most good discussions wind up quoting someone famous, so here’s mine to get us started:

"Everything should be made as simple as possible, but not simpler."
-Albert Einstein

As geeks, we are drawn to adding layers of abstraction. Maybe it’s because it makes us feel smart. Maybe it makes us feel clever. Maybe we think we are better developers because we are engineering for the future. Maybe we are trying to impress our colleagues by showing that we "follow best practices".

Why do we add layers of abstraction? Well, we tell ourselves it’s to make future changes easier. That makes sense. But too often, the scenario for the abstraction layer becoming worthwhile is contrived beyond practical reality. When that happens, you have become cargo cult programmer.

So we add a layer of abstraction to our code under the guise of making it more extensible, usually without fully considering the key consequence of layers of abstraction: increased complexity. More layers to dig through to find problems. More code to maintain. More code to document. More code for the new guy to understand before he can be productive.

Beyond the human implications of having more complexity, don’t forget it’s harder on our servers too, because those extra layers of code also take more CPU cycles to calculate.

My suggestion? When you add a layer of abstraction, make sure that it’s needed in the foreseeable future, and the scenario for it’s use is very likely and thought through, otherwise, skip it and keep things simple.

YAML is a syntax for configuration files.

Huh?

Why on earth do we need another technology for config files?

I first ran across YAML while working with Symfony. YAML’s use in PHP is especially troubling, because PHP has a built in function for parsing config files, parse_ini_file(). This config file syntax is the same as the php.ini file, so it is well known by all PHP system administrators. It’s human readable, supports basic name/value pairs, and allows for comments.

It’s also high performance, since it’s a built in function, config files are parsed with the speed of C instead of text processing with PHP. This may not matter much for Joe Bloe’s blog website, but when you have lots of users, this makes a difference.

From what I saw with it’s use within symfony, everything could have been easily done using parse_ini_file(), and to make matters worse, when it was noticed that there where performance problems with parsing YAML, the Symfony authors decided to add a caching layer. Great. More complexity. This violates the number one rule of good software: Simplicity.

The Symfony authors should have used an existing technology that was built into PHP for handling config settings. Don’t make the same mistake.

Then I looked at wikipedia for the definition of REST, and it let me down. It told me that it stood for Representational State Transfer, about the fundamentals of the idea, as it relates to Roy Fielding’s doctoral dissertation, but I didn’t find it very helpful. I was left asking:

How does this apply in the real world?

I don’t know about you guys, but for me, when a technical exploration winds up centered in academic trivia, it’s frustrating. I’m always looking for the real world implications.

I kept looking beyond wikipedia, and I now understand it’s usage with in Yahoo!, and I think it has potential implications for all - so I’ve decided to share my findings.

Here’s how REST can be used in the real world (at least my version).

Take a typical CRUD app that has to do the following:

CREATE
READ
UPDATE
DELETE

Overlay this on a typical web CRUD framework, and you have a resulting hypothetical file structure that looks like something like this:

CREATE /people/create?username=smartguy&pass=geeksforfun
READ /people/edit?id=323
UPDATE /people/update?id=323&username=smartguy&pass=geeksforfun
DELETE /people/delete?id=323

This should look familiar to all geeks. It can be handled with separate files for the different actions, or use an MVC framework with a controller that intercepts the call and redirects it to a specific action.

Now, consider the built in HTTP methods GET, PUT, POST, DELETE. We all know GET and POST. PUT and DELETE are less common, but they’ve existed since HTTP 1.1. Notice the following associations:

CREATE -> PUT
READ -> GET
UPDATE -> POST
DELETE -> DELETE

Some of you may have just had the light bulbs start to flicker. Next step. Let’s put these together! While we are at it, shorten the urls in the name of SEO. Let’s take a look at our new url structure:

CREATE - PUT /people/
READ - GET /people/323
UPDATE - POST /people/323
DELETE - DELETE /people/323

This is the first 1/3rd of REST. Using an appropriate HTTP request methods to trigger the corresponding CRUD action. Your accepting application should be checking the request method (which is available
in PHP as $_SERVER[’HTTP_REQUEST_METHOD’]) and use that to fork the appropriate action.

Now, the second component of REST is using HTTP status codes for error handling. We already have plenty of status codes built into HTTP. Let’s use them. Here’s a few samples:

200 if it worked.
403 for permission denied.
304 for not modified.
404 for an invalid id
50x for errors

More lightbulbs?

The third component of REST is handling different representations of the data. This is accomplished by specifying an extension on the request.

GET /people/323.txt
GET /people/323.csv
GET /people/323.pdf
GET /people/323.xml

The above calls would all send back the same data, but in different formats based on the extension.

And the fourth component; and this one is key; is that the same methodologies are used by all the data types.

Ie.
GET /people/323.txt
GET /stuff/323.txt
GET /things/323.txt
PUT /stuff/
DELETE /things/323.xml

So you have consistency amongst all of the data elements. They all accept a common set of actions. so that all the web services deal in the same way.

In summary, to build a RESTful web service, it should:
1) Use the GET, POST, PUT, DELETE request methods defined by HTTP to trigger the various actions
2) Take advantage of the extension that the request is using to determine the content type of the response (xml, csv, txt, jpg, etc)
3) Use HTTP status codes (200, 304, 400, 405) to indicate the status of the operation.
4) Be simple and consistent. Implement same methods for CRUDding across all data elements.

Interesting model for web services.

Some may be asking. Isn’t this just SOAP or xml-rpc? Kinda, in that they are all methodologies for implementing web services. REST relies more on HTTP methods and status codes instead of XML for triggering actions.

Additional reading that I found useful:
http://doc.opengarden.org/Articles/REST_for_the_Rest_of_Us
http://webservices.xml.com/pub/a/ws/2003/09/30/soa.html
http://www.xfront.com/REST-Web-Services.html
http://tech.groups.yahoo.com/group/rest-discuss/messages

-Nick