The Computer Guy

XFS PVR/DVR maintenance

Last year, I purchased an HDTV tuner card and started using my computer as a Personal Video Recorder, or PVR. Basically, a small program records a tv show to the hard drive. This did quite well until March when it would stop recording after 5 seconds to 15 minutes for no apparent reason.

After about a month of research, I had ruled out just about everything under the sun. Then, it hit me … the file system might need defraging. The file system I am using is XFS, and they can get fragmented over time, especially if you have big files (HD-MPEGs) and little free space.

The command to defrag all mounted (and keeps them read-write) XFS filesystems is:

xfs_fsr -v

After running this, shows started recording better, but still stopped. So, I wanted to see just how fragmented everything was. I found this command:

xfs_db -r /dev/sda3

That starts an interactive XFS debug session for sda3, my XFS partition. Three useful commands are freesp, frag and quit. frag reported over 15% fragmentation and freesp reported a lot of small blocks and very few large blocks.

Finally, I got to the heart of the matter: defrag programs do not operate efficiently with less than a certain amount of free space. My drive had 20GB free out of 250GB total. I deleted a few HD shows to bring it up to 45GB free and ran the defrag command again.

Voila! Less than 6% fragmentation and plenty of free big blocks. My recordings are almost flawless now. At least they don’t stop for no reason. So, if you’ve got an XFS file system:

• make sure to run the xfs_fsr command regularly
• if you’re filesystem is still fragmented, burn/remove some junk and repeat

HDTV using MPlayer

In my desktop computer, I’ve got an nvidia GeForce 6150 PCI-E video card. It has a built-in mpeg2 decoding helper in it. Using Linux, this is called XvMC, or Xvideo Motion Compensation. I finally got it partially usable.

I’m currently watching the news on WJBF, the only channel I get b/c I haven’t fully setup my antenna. Using xvmc is fairly simple:

mplayer -vc ffmpegmc12 -vo xvmc -framedrop dvb://WJBF-DT

I add -framedrop to keep the video and audio sync’ed.  Mplayer will rant about waiting for retrace. Using OpenGL is actually better, for me at least.

mplayer -vo gl2 -framedrop dvb://WJBF-DT

It dropped 14 frames at the beginning, but hasn’t dropped another in 10 minutes. XvMC would have crapped out by now. A/V sync is still perfect. I’ve even got it full screen and it’s resizing from 1280×720 (720 HD resolution) to 1440×900.

X11 Forwarding

A few days, I moved my desktop computer out into the living room. The computer has a tuner card that can receive digital broadcasts. I don’t own a TV, so this is the closest I’ve got to one. I still have to set up lircd before I can use a remote, but I wanted something a bit easier

Today, I discovered a new trick … SSH X11 Forwarding. Now, I can play video on my desktop right from my netbook. It’s wonderful and very simple!

In the following instructions, the “A” refers to the computer playing the video and “B” refers to any other computer.

1. Make sure “X11Forwarding Yes” is in your sshd_config file on computer A. Restart sshd if you had to uncomment/add it.
2. On B, run `ssh -XC A`
3. In the ssh session, type `DISPLAY=:0.0`.
4. Now run mplayer or any other video player in the  ssh session and it will play on computer A.

If you want  to have a program run on computer A and use the screen on computer B, the process is very similar:

1. Make sure “X11Forwarding Yes” is in your sshd_config file on computer A. Restart sshd if you had to uncomment/add it.
2. On B, run `xhost A`
3. On B, run `ssh -XC A`
4. Now run mplayer or any other video player in the  ssh session and it will play on computer A.

Step 2 authorizes use of the current display by computer A and only needs to be run once. Also, note that the display variable doesn’t need to be set when using B’s screen.

Mplayer works wonderfully when playing on my desktop, plus I control it via the ssh session. It has tons of keyboard shortcuts. The ones I use the most are space for pause and the arrow keys for skipping around.

PVR Update

In anticipation of TV shows coming back on, I’ve been tweaking my PVR scripts. These record shows from the digital airwaves right onto my computer. I’ve separated the script into two parts: recording and encoding. Each is a simple one line shell script that can be called easily from the command line or from a cron. The record.sh script is:

/usr/bin/mencoder dvb://$1 -ovc copy -oac copy -o "$2.mpeg" -endpos $3

All this does is copy the video and audio streams directly to the disk. The three parameters are  the station name, as stated in ~/.mplayer/channels.conf, the filename and the end position. This end position can be the number or seconds to record, or a time format of hh:mm:ss. This makes recording fairly simple within the cron.

The encode.sh script contains:

/usr/bin/mencoder -vf scale=640:-2 -ovc xvid -xvidencopts bitrate=1700:threads=2 \
 -oac copy "$1" -o "$1.avi"

This, slightly more complex use of mencoder, will scale the video to 640 wide, the height calculated to keep the aspect ratio. The video is encoded into xvid at 1700kbps using 2 threads. To speed up the encoding process, the audio is not re-encoded, but copied as is.

You will need to play with the number of xvid threads to find the best fps. I have a dual-core and it encodes the fastest with 2 threads. It does need to be an integer, and most dual-cores will work best at 2.

Then, two lines are required in the cron to record and encode the program:

59 19 * * 1 /dvr/record.sh WAGTNBC /dvr/shows/heroes 01:03:00
05 20 * * 1 /dvr/encode.sh /dvr/shows/heroes.mpeg

Mencoder Cheat Sheet

Mencoder is a video encoder for linux. It is part of the mplayer package which also includes a video player. This tool is quite versatile and can do pretty much any kind of video processing all from the console.

As with any linux tool, there a tons of bells and whistles. This makes the learning curve kind of steep, but with a bit of determination anyone can make sense of it. Without further ado, here’s the cheat sheet:

PVR Cron

For about a year now, I’ve been playing around with my digital tuner card. It wasn’t until I turned off the cable that I have a need to use it. Using some cool linux tools, I’ve made a script to record HDTV broadcasts to my computer. It is a work in progress, but here’s what I’ve got so far.

The Tuner Card
The tuner card is DVB based on a Conextant chipset, so the first step was to get my kernel to make the card usable. A quick check will show if the driver is loaded:

dmesg | grep dvb

The Tools Required

• dvb-atsc-tools
• azap
• ffmpeg

Channel Scan
Scan for channels using:

dvbscan /usr/share/dvb/atsc/us-ATSC-center-frequencies-8VSB > ~/.azap/channels.conf

Edit the file ~/.azap/channels.conf to make sure the channel names are correct. Your base frequencies file may be in a different location, but it is usually under /usr/share.

Iteration 1: Crontab Recording
At first, I used only the crontab to record. Here’s an example:


24 12 * * * /usr/bin/azap -c /home/dvr/.azap/channels.conf -r WRDW-HD
25 12 * * * /bin/cat /dev/dvb/adapter0/dvr0 > /data/dvr/young-restless.mpeg
35 13 * * * /usr/bin/pkill cat
36 13 * * * /usr/bin/pkill azap
37 13 * * * /usr/bin/ffmpeg -i /data/dvr/young-restless.mpeg -s 1024x476 -vcodec libxvid -b 1600000 -acodec copy /data/dvr/y-r-friday.avi


This is a very ugly solution with lots of cracks. For instance, if I were running cat from a console when /usr/bin/pkill cat were running, it would die. Heaven forbid another processes is using cat when that runs. Also, I had to change the name of the ffmpeg output file every day.

Iteration 2: Cronable Perl Script
This script does pretty much the same thing as the above 4 lines in the crontab does. This means you don’t have to write 4 lines in the crontab for each recording, just 1 line. Also, the file name is appended with the date in yyyy-mm-dd format.


#!/usr/bin/perl

#Does someone need a reminder?
if ( $#ARGV != 2 ) {
print “Usage:\n”;
print “record.pl \n”;
exit;
}

#Creates a random 16 character (a-z) string
sub randstr {
my @chars=(‘a’..’z');
my $res = “”;
for(my $i=0;$i<16;$i++) {
$res .= $chars[rand($#chars)];
}
return $res;
}

#Grab the command line args
my ( $channel, $length, $finalFileName ) = @ARGV;

#Temporary mpeg filename
my $tempFileName = randstr();

#Add date to final filename
$finalFileName .= “-”.`date +%Y-%m-%d`;
$finalFileName =~ s/\n//;

#Start Azap in the background
print “Starting azap\n”;
system( “/usr/bin/azap -c /home/barry/.azap/channels.conf -r $channel >/dev/null 2>/dev/null &” );
sleep 5;

#Start cat in the background
print “Starting cat\n”;
system( “/bin/cat /dev/dvb/adapter0/dvr0 > /data/dvr/$tempFileName.mpeg &” );

#Sleep the required seconds for the show to record
print “Recording for “.(60*$length).” seconds…\n”;
sleep 60*$length;

#TODO: Remove pkill, as it may cause problems
print “Killing cat and azap.\n”;
`pkill cat`;
`pkill azap`;

#Resize & Encode to XVID using ffmpeg
#ffmpeg sometimes stops working b/c of bad mpeg data
#TODO: Replace with mencoder
print “Encoding…\n”;
`/usr/bin/ffmpeg -i /data/dvr/$tempFileName.mpeg -s 1024×476 -vcodec libxvid -b 1600000 -acodec copy /data/dvr/$finalFileName.avi`;

#Remove the temporary mpeg file
`rm $tempFileName.mpeg`;

print “Done!\n”;

I know it’s not the most elegant of perl scripts, but it gets the job done. Here’s a sample cron:


25 12 * * * /home/dvr/record.pl WRDW-HD 70 young-restless


As you can see from the TODO comments, I continue to tinker with the script. When I make a good development, I’ll post it. If you have any suggestions, feel free to post a comment or contact me.

Cron Jobs

Cron jobs are the Linux/UNIX world’s scheduler. Many people, myself included, have a hard time understanding crons. After all, setting one up is very simple, if you know what you are doing.

The Crontab
“crontab” is the utility you use to setup a cron job. Type “crontab -e” from the command prompt to edit your list of crons. It’s just a textfile with a special format. Each non-blank line is a cron job. The format of each line is 6 space separated fields. Here is what each field defines:

1. Minute
2. Hour (Military Time)
3. Day of the Month
4. Month
5. Day of the Week (0 is Sunday, 1 is Monday, etc.)
6. Program to run w/any arguments (may include spaces)

The Date/Time fields may be a specific number, a “*” meaning anything, a “*/x” meaning every x hours/minutes/etc., or a comma separated list of any or all of the above.

So, if you wanted a certain program to run every 30 minutes, you could say it any of these ways:

*/30 * * * * /path/to/program arg1 arg2
0,30 * * * * /path/to/program arg1 arg2
12,42 * * * * /path/to/program arg1 arg2

Use Absolute Paths
If you are writing a script to run as a cron, you must make sure your script uses full paths or it will throw an error. Yes, the script will execute without error when run from the command line, but the cron will crap out.

Examples

0 2 1 * * /home/apache/rotate-logs.sh

Runs the rotate-logs.sh script the first of every month at 2AM.

0 17 * * * /home/oddjob/send-margarita-time-email.sh

Runs the send-margarita-time-email.sh script at 5PM every day.

0 17 * * 1,2,3,4,5 /home/oddjob/send-margarita-time-email.sh

Runs the send-margarita-time-email.sh script at 5PM Monday – Friday.

0 4 * * 1 /usr/sbin/mysql_dump > /backup/mysql/backup-`date "+%Y-%m-%d"`

Performs a weekly backup of the mysql database Mondays at 4AM.

0 0 1,15 * * /home/accounting/payroll.pl

Runs payroll.pl on the first and fifteenth of every month at 12 midnight.

Points to remember

• Use full paths in scripts
• The crontab is only for the current user

Vim quick start

Vim is a open source text editor.  I fell in love with it a few years ago.  It runs on Linux, Windows, Macs, and others.  The interface is pretty much the same across the board. It really is full-featured.  It highlights almost any file syntax from PHP to python to httpd.conf.

One major barriers to its popularity is its learning curve.  Vim was designed to be used entirely by a keyboard.  To start, download it and install it.  If you’re running linux, it should be preinstalled.  Once it is installed, run it by running vi. When you run it, you can only scroll.

Press the i key or the insert key to go to insert mode and you can type all day long. Press the escape key to exit insert mode.

You can do a quick search by pressing forward slash and typing your search. To repeat the last search press the foward slash then enter.

All the other basic commands start with a colon.  Here are a few to get you started:

• :w — write the file
• :sav filename — save the file to filename
• :q — quit
• :n — go to the next file
• :N — go the the previous file
• :wq — write the file and quit
• :q! — quit now!
• :wn — write the file andd go to the next file

There is a key difference between “:q” and “:q!”.  The first will not quit if the file has been changed or there are more files to edit.  The second will quit immediately.

There are  a ton more features than what is listed here.  When you are ready to continue learning vim, type :help for a help screen.  Type :q to exit the help.

Happy editing.

Corporate Linux

Large corporations are shoveling cash into Microsoft by buying their software which requires minimal setup for running a large network of corporate machines. Large organizations think it’s cost effective versus Linux, and they’re probably correct if someone in the corporation develops software using GPL software as a base for the project. How many large corporations do this? Very few.

Most corporations do not use Linux because they think the change over will cost them a lot of money. While this may be true for almost any large bank of computers, the cost in the long-run is much better with Linux than Windows. Cost is only 1 benefit of changing over to Linux.

Another benefit is security. Almost every program designed to run on Linux is created by the community, so bugs and security holes are found and fixed quicker than Microsoft can do it. The updates can be pushed using a free program found on the internet and installed on every machine, or individual ones.
The change over does require some user training, but it’s pretty simple to learn. At my work, we use a lot of free software that is available for Window and Linux; however, we run Windows on all our machines, except the web servers which run Linux.

Most people will use an app for business, another for email, and a basic web browser. If the business app was written for Windows, the change over cost can be astronomical, but if the app is web based, the change over cost is minimized. Email is email and the web is the web, no matter what program you use to access it.
Watch out though! Sarbanes-Oxley entangles Linux with a lot of red tape. This is mainly due to the fact that the CEO is personally responsible for all reporting of intellectual property information as well as all financial information.

If a corp is simply using it as an OS, the red tape is only a couple of levels thick. If anything is modified, the layers of red tape can amass pretty quickly. This is due to the wording of the GPL, the most common open source software license for Linux programs.

I asked the tech guy at my work if we could do a change over and he said it would be a bunch of red tape, mainly due to the rewriting of policy and building desktop OSs from scratch. I agree with the building the OS from scratch, which I have done before and is pretty simple once you’ve done it a few times. Policies need to be rewritten anytime an OS change over takes place, so go ahead and make the policies!

Anyways, for any corporation just starting out, go with Linux. It’s a lot cheaper than Windows, $0 vs. $300. It can be a pain to set up if you are not familiar with it, but you’ll save $300 per machine. That’s enough to pay someone for 30 hours of work at $9.29, $0.71 is taken off the top for business paid FICA taxes. Plus if you buy all the same machine, once it’s setup on one, you can simple copy hard drives to get it on all the other machines.

Death of Windows XP

Windows XP will be pulled from the shelves in less than 3 months time. The official finale will be observed June 30, 2008. Many people are up in arms over this. There is even a petition to keep XP on the shelves. As the date nears, I am advising people who will be in the market for a new computer between now and 2010 to try Linux.

Try it, that’s all I ask. Linux has advanced dramatically from it’s conception in 1991. There are several distributions, or flavors, that offer a LiveCD or LiveDVD. A LiveCD/LiveDVD is simply a copy of Linux that runs directly from a burned CD/DVD. It installs nothing to your computer, so you aren’t forced to do anything to your computer except drop the disc in and reboot.

Many of these LiveCDs allow you to install Linux after you have tried it. For a newbie, or noob(pronounced the same way), Ubuntu is your best bet. Anyone with a blank CD-R, a CD burner, and a high speed internet connection (or a friend with high speed) can do it.

Linux is different from Windows; however the KDE desktop environment is very similar to Windows and comes with a Redmond* Theme that looks almost exactly like Windows XP. The programs are called different names, but you can enjoy a lot of the same programs, like Firefox.

I know, some of you might be saying, “What about my games?” They’ve got most of them covered too. There’s a Windows Emulator** called Wine. There’s also a program call PlayOnLinux which sets everything up for many many Windows programs, including my favorite game, WoW.

So, give it a try! You might be sorely disappointed, but I doubt you’ll be that disappointed.

* Microsoft is based in Redmond, WA. Hence the name.

** An emulator is a program, or set of programs, that allows you to run a different system of software, such as Playstation or Windows.