My First Lesson - production.log?

Okay, so right off the bat I'm not going to do what I said I was going to do, but hear me out. I haven't actually made this mistake before but, true to the spirit of avoiding self-inflicted wounds, this one was self-inflicted, so it counts!

Standard scenario: try to do something as the admin in a production application and discover a bug. "Hmmm, wonder why that happened -- let's check the log." Jump into a terminal window, tail -f production.log, repeat the faulty action and...what the hell, why is nothing being written to the log?

Well, almost nothing. While staring at the log the cron job that runs ar_sendmail wrote to the log just fine, but the application itself wasn't, regardless of what I was doing with the app in a browser.

Without the benefit of time travel and 20/20 hindsight I couldn't check this blog, so I started searching for an answer. The probable culprit in most of the relevant posts (and there weren't many, which surprised me) was a permissions problem on the log file itself.

Not to toot my own horn (so to speak [as if a Viper pilot is going to read this]), but I don't limit myself to being a n00b in just one area. In addition to being wet behind the ears in Rails I'm also a rank (as in, "Man that stuff stinks! It's rank!") beginner with Linux, bash, chmod and a whole bunch of other stuff that, frankly, I don't know well enough to even know that I don't know it.

But enough about me -- back to the topic at hand. I took a look at the permissions of all the files in the /log directory (using ls -l) and quickly noticed that the permissions for production.log didn't match the development.log or mongrel.log files.

Being razor sharp I decided to make 'em the same. But how? Meet our friend chmod. Not being an expert here I just copied what I saw in some of the stuff I found online:

chmod 0666 production.log

Now the permissions at least matched the other files in terms of rw-rw-rw-. I tried the app again but still, nothing was written to the log.

Going back to the /log directory with another ls -l it dawned on me that the owner and group for production.log (chris sudo) were different from the other files (root root). But how to change that?

Meet our other new friend, chown, cleverly named for what it does, "change owner". A bit more of the google and I found the syntax (I know, I know, how about man chown? I'm a newbie, remember?):

chown root:root production.log

Now the permissions, owner and group all match the other non-FUBAR'd files. A quick click in the app and, thankfully, it's working again.

To quote David Byrne, "but, how did I get here?" Again, in the true spirit of this blog, I was too smart for my own good. I'd seen numerous "best practices" articles about not letting the production.log file get too huge, and all suggested using the *nix logrotate command to customize how the log file gets truncated, copied, compressed or whatever other stuff you want to do to it.

Wanting to join the club of People Who Produce Professional Rails Apps (yet really don't know what the f*ck they're doing), I dived right in, head first (watch out for rocks). Here's the logrotate.conf script I came up with (stored in ~/cfg):

/home/me/myapp/shared/log/production.log {
rotate 4
weekly
create 0666 root root
compress
notifempty
missingok
dateext
nomail
}

Note the 0666 root root after create. That wasn't there before and was, I'm pretty sure, the source of the problem. When the production.log file was being re-created with plain, unadorned create it didn't have the appropriate permissions, owner or group to allow the rails process to write to it. Or at least, that's my newbie suspicion.

I'll know for sure that it works once the log rotates and actually functions properly. If I hadn't run out of steam last night after getting this far I would have done the truly professional thing and figured out how to run my logrotate.conf file on the spot to be sure.

But, as is the nature of the software development business, that's a rabbit hole I may run down some other time, undoubtedly in the wee hours, after a long day, with a deadline looming, and a tiny voice in my head saying, "Haven't I seen this before?...."

Time for bed -- I've got to spend some time above Armstrong's Line tomorrow morning.

Lessons Learned (and Mistakes Repeated) - Adventures in Ruby on Rails

Tell me if this sounds familiar: you run into some totally goofy problem with your Rails application, a problem that you're completely confident you've seen (and conquered) previously after a few hours of using the google on the internets, finding a solution, then spending several more hours tweaking your app, the deployment or one of the other five-dozen moving parts in the typical RoR app.

But the elation lasts about 5 seconds because you realize:

• You didn't write down how you fixed it last time, or
• You didn't bookmark the site where you found the answer, or
• You did bookmark the site, but have no idea what the hell the bookmark is called, or
• You did bookmark it, you do find the bookmark, but the site's gone belly-up

Yeah, I thought so. Well, this is my attempt to save myself, and anyone else who stumbles across this blog, from suffering so many self-inflicted forehead slaps.

Being a relative newcomer to Ruby on Rails, and despite my expensive attempts to get smart (measured in money spent on books, Peepcode screencasts and PDFs and, actually, really good training) I somehow still manage to make the same dumb mistakes regularly.

The band-aid I'm going to place over that gaping wound of jackassery is the listing here of those foul-ups and how I fixed them. Perhaps the next time I get that deja vu feeling all over again I can spare my cranium an open-handed "Dooh!".