TheFreyers.net

This page isn't a comprehensive guide to rsync¹⁾. Instead, this is my quick reference to synchronizing a remote directory with a local one using rsync.

First, the scenario: Machine 1 contains a constantly-changing set of files that I want to sync to a different directory on Machine 2. I could login to either machine to complete the rsync, but I'll login to Machind 1 and “push” the files to Machine 2. The scenario is best illustrated like this:

The files I'm syncing are TV recordings. They won't be modified once they reach their destination (although rsync could handle that), but the set of files on Machine 1 changes slightly each day. The most important thing is to add and delete files from Machine 2 to keep it in sync with Machine 1.

Here is the statement that causes the sync to happen:

cd /video
nohup rsync -r -a -v -e "ssh -l myIdOnMachine2" --delete tvrecordings/ Machine2:/data/mythtv/recordings/

Its a powerful command:

-r = recursive
-a = archive (copying owner, group, permissions, etc…
-v = verbose (optional)
-e = name of remote command to use
–delete = delete unneeded files from destination directory

Its critical to note that the source directory is listed first, and the destination directory is synced to it. If you reverse these, the process will work in reverse–syncing your source directory to the destination.

¹⁾ If you need that, please see the man page

Linux

Chris Freyer — Mon, 08 Mar 2010 14:37:07 -0500

Linux

I started using Unix professionally in 1993 and discovered Linux in 1994. My first distribution was CD-based and came with an X11 desktop. It came with plenty of documentation from the Linux Documentation Project. It was exactly the right information at the right time in my career to help me really learn Unix.

Over the years I've had the fortune of installing and using many great distributions. Of special note are these distributions:

Yggdrasil_Linux - plug-and-play, first live CD distro, being sold at shareware shops
Slackware - widely-available through book publishers
SuSE - fantastic installer, YaST management tool, CD/DVD-based distribution model¹⁾
Debian - apt-get and dpkg tools, online repository, support of many architectures
Mepis - big advances in desktop usability, based on debian
Knoppix - great for troubleshooting and debugging, based on debian
Gentoo - build-from-source mindset, ebuild tools, online source code repository
RedHat - RPM package manager, efforts to 'go corporate' with Linux
Ubuntu - installer, usability, desktop maturity, based on debian

Notes and Shell Scripts

Here are some notes and shell scripts that I've written.

mp3_organizer rewrites a flat directory of long mp3 filenames into a meaningful directory structure
smart sends email updates about the health status of hard drives
visitors creates a visual graph of web traffic
backup very simple cross-site backup script.
fail2ban stops bots and script kiddies in their tracks.
cpuspeed shows CPU speed every second
recursivecopy small script to do a selective recursive copy
Runlevel tools
Logfile tricks
why_rpm_bugs_me
Dealing with open files
How long has a process has been alive?
wma2mp3 - a small script to convert Windows WMA files to MP3 format
Testing UDP availability to a remote host
How To Backup an Oracle XE Database
How To Restore an Oracle XE Database

How-Tos

LVM
SSH Tunneling
Screen - a text-mode window manager
Creating a 4-Disk RAID Array for large amounts of storage
How To Use rsync

¹⁾ this was before the popularization of high-speed internet access

How To Restore an Oracle XE Database - created

Chris Freyer — Wed, 03 Mar 2010 09:03:04 -0500

How To Restore an Oracle XE Database

Restoring a database is fairly easy if you have a backup file that was made with the procedure outlined on the How To Backup An Oracle XE Database page.

I'm assuming you have already transferred the backup file to the server where the restoration will happen, and the file is named ora_backup.tar.gz. The first step is to extract the archive and copy the files to the Oracle home folder:

[me@myhost ~]# /etc/init.d/oracle-xe stop
[me@myhost ~]# mkdir ora_backup
[me@myhost ~]# cp ora_backup.tar.gz ora_backup/
[me@myhost ~]# cd ora_backup/
[me@myhost ~]# tar xzf ora_backup.tar.gz
[me@myhost ora_backup]# cd usr/lib/oracle/xe/
[me@myhost xe]# cp -rf * /usr/lib/oracle/xe/

The copy process will take several minutes, depending on the environment. When finished, all the database files, the flash recovery area files, and the Oracle SP files will have been overwritten with the files from the backup.

At this point, you might think you could simply start the database. Unfortunately this isn't the case. Oracle stores some of the system configuration data inside the database itself. So you have to re-run the XE configuration script again to get the new configuration data into the database files.

Take a look at the /etc/sysconfig/oracle-xe file. It contains the answers that were given during the original run of the configuration script. You'll need to answer the questions the same way during the next run or your database might not operate the same as before.

[me@myhost ~]# /etc/init.d/oracle-xe stop
[me@myhost ~]# cd /etc/sysconfig
[me@myhost sysconfig]# mv oracle-xe oracle-xe.original
[me@myhost sysconfig]# /etc/init.d/oracle-xe configure
 
Oracle Database 10g Express Edition Configuration
-------------------------------------------------
This will configure on-boot properties of Oracle Database 10g Express
Edition.  The following questions will determine whether the database should
be starting upon system boot, the ports it will use, and the passwords that
will be used for database accounts.  Press <Enter> to accept the defaults.
Ctrl-C will abort.
 
Specify the HTTP port that will be used for Oracle Application Express [8080]:8888
 
Specify a port that will be used for the database listener [1521]:
 
Specify a password to be used for database accounts.  Note that the same
password will be used for SYS and SYSTEM.  Oracle recommends the use of
different passwords for each database account.  This can be done after
initial configuration:
Confirm the password:
 
Do you want Oracle Database 10g Express Edition to be started on boot (y/n) [y]:y
 
Starting Oracle Net Listener...Done
Configuring Database...Done
Starting Oracle Database 10g Express Edition Instance...Done
Installation Completed Successfully.
To access the Database Home Page go to "http://127.0.0.1:8888/apex"

Note the database has been started as part of the configuration script, and it offers a local URL for the database console. You should login and check your system to ensure all data is present.

Wiki Mindset

Chris Freyer — Mon, 01 Mar 2010 11:43:10 -0500

Wiki Mindset

To work in a wiki environment, it helps to understand the philosophy first.

A wiki is essentially a book that is created by a group of editors. One of the editors acts as the administrator. The administrator creates the book's structure–its table of contents. Then the editors begin creating and editing documents that are linked to the table of contents and to each other. Its easy to see how free editing can cause conflicts. So here are a few principles…

Understand the basics

Hyperlinks are one color if they have a page associated with them, and another color otherwise. The default colors for Dokuwiki are green and red (respectively), but this can be changed by using a different wiki theme.
Always be sure you're logged in. Some wikis don't require you to login before editing, but doing so will ensure the system can communicate with you via email about important events.
Hit the “Subscribe Changes” button on any page to get emails when the page is edited
The “Recent changes” button or link shows the revision history of the current page
The “Search” function works really well. Pages can be searched as soon as they are saved. And you can use advanced techniques like regular expressions.
The bread crumb trail at the top of the page helps you remember how you navigated to a page

Use your signature

When editing a document, clicking the right-most icon in the button bar will insert your signature. You can use your signature to:

sign your work
claim an empty document as yours (for future editing)
provide feedback to others (in the form of a question or revised text)
etc…

Important realizations

A wiki requires you to create a hyperlink first, then click on it to edit a new document.
You will work faster if you focus on the content first, not how it looks.
The wiki creates a table of contents automatically (based on H1, H2, and H3 tags).
You can edit a section of a document by clicking the EDIT button next to that section.
The file upload feature is a “picture frame” icon in the button bar of the edit page

Things to Avoid

There are some things that editors should not do:

avoid big edit sessions. Make many small edits + saves instead.
avoid overly-complex page designs.
Don't expect to have complete control of anything.
Don't undermine others. Their work is just as important as yours
Don't upload files if their content needs to be edited over time. Paste their content into a new wiki page.

Things to Do

Equally important, there are things editors should do:

Use a consistent style. H1 for page headings, H2 for subject headings, etc…
Create links in your documents to other documents (this builds cohesiveness)
The “Preview” button is your friend. You'll understand soon enough.
Save often. There is no penalty for saving many times.
If you delete information from a document, say so in the edit summary when you save. You'll be able to find the deletions easier later on.

Adding iLBC to Asterisk

Chris Freyer — Thu, 25 Feb 2010 19:04:04 -0500

Adding iLBC to Asterisk

The Internet_Low_Bit_Rate_Codec is a freely-available stream processor that works really well with human voice compression. It typically delivers clear voice signals in ~16kb of bandwidth. For that reason, it has become very popular in Asterisk implementations. Unfortunately there have been license disputes in the open source community and the Asterisk maintainers have removed iLBC from the source tree. But you can still use it if you know a few tricks.

Build from Source

Depending on your distribution, you might need to build Asterisk from source. I'm using Red Hat Enterprise Linux 4, and Asterisk isn't one of the available packages. So here's how I did it. (I'm not going to detail how to install the kernel source, kernel headers, etc… See your distribution's instructions for that).

  cd /usr/src
 
  tar xzf asterisk-1.4.25.tar.gz
  cd asterisk-1.4.25
 
  # the following lines overcome a compile problem in 
  # RedHat Enterprise Linux 4 where the iLBC source isn't 
  # found by make, even after it is downloadeded.  
  # Your distro might not need this, so skip down as needed
  # get the ILBC source
  cd contrib/scripts/
  ./get_ilbc_source.sh
  cd ../..
  cd  codecs
  #the codecs/ilbc directory is empty, so make it a symlink to 
  #the contrib/scripts/codecs/ilbc directory
  rm -rf ilbc
  ln -s /usr/src/asterisk-1.4.25/contrib/scripts/codecs/ilbc ilbc
  cd ..

Then…

  ./configure
  make menuconfig

The menuconfig option gives a menu-based approach to configuring the source. ILBC is disabled by default, so we need to enable it. Here is what the main screen looks like:

               **************************************************
                   Asterisk Module and Build Option Selection
               **************************************************

                               Press 'h' for help.

                              1.  Applications
                              2.  Call Detail Recording
                              3.  Channel Drivers
                         ---> 4.  Codec Translators
                              5.  Format Interpreters
                              6.  Dialplan Functions
                              7.  PBX Modules
                              8.  Resource Modules
                              9.  Voicemail Build Options
                              10. Compiler Flags
                              11. Module Embedding
                              12. Core Sound Packages
                              13. Music On Hold File Packages
                              14. Extras Sound Packages

Go into option 4 and enable codec_ilbc, like this:

               **************************************************
                   Asterisk Module and Build Option Selection
               **************************************************
  
                               Press 'h' for help.
  
                              [*] 1.  codec_adpcm
                              [*] 2.  codec_alaw
                              [*] 3.  codec_a_mu
                              [*] 4.  codec_dahdi
                              [*] 5.  codec_g726
                              [*] 6.  codec_gsm
                              [*] 7.  codec_ilbc
                              [*] 8.  codec_lpc10
                              XXX 9.  codec_speex
                              [*] 10. codec_ulaw
  
                       Adaptive Differential PCM Coder/Decoder

Then backout and goto option 5 to enable format_ilbc

               **************************************************
                   Asterisk Module and Build Option Selection
               **************************************************
  
                               Press 'h' for help.
  
                              [*] 1.  format_g723
                              [*] 2.  format_g726
                              [*] 3.  format_g729
                              [*] 4.  format_gsm
                              [*] 5.  format_h263
                              [*] 6.  format_h264
                              [*] 7.  format_ilbc
                              [*] 8.  format_jpeg
                              [*] 9.  format_ogg_vorbis
                              [*] 10. format_pcm
                                  ... More ...
  
  
                        Raw iLBC data

After that, I recommend scanning through the other options to see if there are other things you need. In particular, options 12-14 contain some helpful soundfiles.

Exit out of the menuconfig and press 'S' to save your options. After that, its a fairly standard linux build process:

make
make install
make samples
make progdocs

The Dog Page

Chris Freyer — Thu, 25 Feb 2010 17:10:25 -0500

The Dog Page

We added a new dog to our family on July 4, 2009. She's an 11-week-old Golden Retriever pup. Our kids named her Sunshine, which seems to fit. This page documents various things about our experience.

Basic Obedience Training

We want our dog to be able to do certain things in our house…like sit, “drop it”, shake, and so on. So we enrolled in Beginner Education classes at Pet Smart.

Week 1

Adjustment takes time. New people + new experiences + new environment = one scared puppy.
Cats don't like surprises. We failed to consult our cats when buying a dog. They now live upstairs, only venturing down for nourishment when necessary. We moved their litter box upstairs as a bribe to keep them from using our carpet as a substitute.
Puppies chew anything. So far its been limited to fingers, shoe laces, and a few kids toys.
Puppies eat anything. Grass and mulch seem to be good substitutes for Eukanuba.
Puppies don't understand boundaries:
- Playing in the front yard or in the street - no difference
- Being muddy outside vs. inside - no difference
Watch for the puppy dance – walking in small circles, tail up, while smelling the ground. This happens right before dogs need to answer the call of nature.
- Diligence will save your floors
- Crate training is your friend
- House breaking takes time

Week 2

Amazing progress in a week. Housebreaking is going well–only mistakes are when we don't follow our routine. Here is what we do.

Go outside immediately after waking up
30-minute walk each morning
- never leave home without a leash.
- always take plastic bags (for cleanups)
- practice learning commands: sit, stay, come
Eat/drink/play
Go outside 15 minutes after eating.
Return the dog to her crate whenever we cannot watch her (work, leaving the house, etc).

This will probably work during the summer while the family is home. Hopefully it will be a long enough time to establish a pattern.

Week 3

Sunshine started showing resistance to entering her create. I bought a large dog pillow and placed it by the fireplace for a day. She loved it and started taking her naps there. The next day I moved it into her crate. Now she enters the crate freely. I guess there's a comfort factor involved.

Looking back to our first attempts at walking Sunshine on a leash, she's made great progress. She resisted fully in her first few days, digging her heels into the ground and fighting the leash regardless of our attempts to praise or encourage her. Now (10 days later), we can walk at a normal pace and she willfully walks with us. She hasn't mastered the art of walking in a straight line yet (understandable) and is frequently underfoot.

We started Sunshine on a wellness plan at Banfield Animal Hospital. Our cats are already on Banfield plans. The plans are a great way to levelize the financial spikes that happen when a pet gets sick or needs vaccinations. Here are my thoughs on Banfield Plans.

We started taking obedience classes (they call them “puppy classes”) at PetSmart. Topic #1 is about crate training. Dogs must be taken outside immediately after being let out of their crate. We'd already learned that lesson the hard way, but it was good to know we're doing the right things on our own. Some dogs get distracted when they go outside and don't do their business. So for those dogs, owners are told to stand in the same place each time, repeat the same phrase, and only play with the dog or praise it when its done. And if the dog doesn't do its business, then it goes back inside and into the crate. The process is repeated until the dog does what it should.

Obedience topic #2 is the “sit” command. We had already worked on that too. Not too hard to encourage Sunshine to sit when you have treats in your hand. Pieces of Milk Bones work well.

Week 4

I forgot to mention previously…Sunshine was diagnosed with a parasite called Giardia after we bought her, so we gave her medicine for the first 2 weeks we had her. Near the end of that period, she developed a deep cough so we took her to Banfield. The vet did several tests and found that Giardia was still present. So due to the timing, we got several meds:

stuff to kill the Giardia
an antibiotic for the cough
a dewormer
a vitamin liquid
ear drops to fight an ear infection

Once we got home, we quickly realized that its tough to force-feed pills to a dog. We went back to PetSmart and found something called Pill Pockets. They are a soft, hollow dog treat. We stuff the pills inside, close the treat, and give it to Sunshine. Mmm…she likes them. She's even started barking in anticipation of getting her treat. How cool is that?

Obedience topic #3 is leash training. Large dogs can be dangerous for kids or non-athletic adults when not well trained. (I have this Farside cartoon in my brain of a dog running down the street, dragging a leash with an arm connected to it, while a man on the sidewalk says to his wife “I told Frank he should train that dog”). So we practice walking at a calm speed. If Sunshine tugs hard on the leash, we stop and pull her back gently while calling her. Her reward is for the walk to continue. We've been walking several times a day for a month now, so Sunshine knows the routine. All is well until she sees a person or another dog, then all her training goes out the window while she practically strangles herself to reach the person/dog and say hello. We're working on that part.

Obedience topic #4 is the “down” command. Surprisingly, this is easy too. We use treats to guide her to the floor while saying 'down'. Right now, it only works if we have a treat in our hand and we physically put it down near the floor.

Week 5

Very unusual…Sunshine woke up on Wednesday morning, August 5, and had more energy than ever before. Its like someone flipped a switch. I think all the medicines & vitamins she's been taking have finally kicked in. Anita took her for a walk in the morning; the kids took her in the afternoon, and I took her in the evening. I took her on the longest walk she's had to date…over a mile. Its not really a “walk”, but kind of a run-walk-sniff-walk. And she still had energy to burn when we got home. Amazing improvement. The same high-energy has been present all week, so we're getting used to that.

Sunshine has learned to stand on her hind quarters this week. She's using that skill to reach counter tops, torment the cats, beg for food, etc. We need a strategy for dealing with that.

Obedience topic #5: “Leave it”. This is a verbal command for making a dog leave something alone. Our trainer tempted Sunshine with a piece of beef jerky. When she went for it, the trainer put her hand in front of Sunshine's chest and pushed her back while saying “leave it”. After a few attempts, Sunshine stopped going for the beef jerky and got a treat instead. This seemed fairly easy in class.

Obedience topic #6: “Drop it”. Similar to the above, but used when a dog has something in its mouth that you want it to release. We gave Sunshine a piece of rawhide and let her start chewing on it. Then we placed a dog treat against her nose and said “drop it”. Something with the strong scent of the treat caused her to forget about the rawhide. Two or three repetitions of this and Sunshine was dropping the rawhide in anticipation of the treat.

Week 6

We started the week by taking Sunshine to the vet for a checkup and vaccinations. The Giardia parasite is gone from her system…finally. She received all of her immunizations (I can't name them all), including rabies.

Obedience topic #7: Sit-stay. Basically we tell the dog to sit, then use any method possible to make the dog sit for longer periods. Goal is 15 seconds. Holding a treat almost directly over Sunshine's head works really well. We were able to reach 10 seconds on the 2nd or 3rd try. Other dogs would barely sit, so we were proud of our dog.

Obedience topic #8: Down-stay. Same as above, but maintaining a down position. Similar tactics are necessary, depending on the dog. We reached 10 seconds on the first try.

Obedience topic #9. Come. Our trainer held Sunshine's leash. We let Sunshine sniff the 3-4 treats we held in our hand. Very strong smell, so we got her attention. Then we ran quickly to a spot about 30 feet away (the running motion grabs most dog's attention). Then we knelt down and called Sunshine to “come”. Luckily for us, she came to us quickly every time. Other dog owners weren't so lucky.

Week 7

No new topics this week…just more reinforcement of the previous lesson. They are difficult lessons to learn, so they merit more time.

A few of our classmates have dropped out by now. Sad situation, but it doesn't surprise me. Some dogs don't respond well to obedience training, and some people aren't focused on doggie discipline and training. So there's a drop in attendance as people become frustrated.

We were told that Sunshine would be expected to sit/say and down/stay for 30 seconds in order to graduate.

Week 8

Graduation week! Sunshine and 3-4 other dogs were there. Our instructor brought several other people to watch (new course attendees, I imagine). The class was challenged with a task, and each owner would take turns showing that his/her dog could perform the task. And the cycle repeated. Our very last task was to keep our dog in down-stay position for 30 seconds. We were able to do that, and even placed doggie treats on Sunshine's paws while she was waiting. Sunshine got her training certificate.

Intermediate Obedience Training

Since we want our dog to be well-trained, we decided to enroll her in intermediate training. She has the aptitude for it, and we (as a family) are continuing to work with her at home. So…off we go into intermediate training.

Week 1

Our new class had only 3 other dogs on the first day–the trainers dog, a German Shepard mix, and a Bull Mastiff (who likes to eat drywall when he's bored). The German Shepard is full grown (2-1/2 years old) but is still highly trainable. The Mastiff is only 10 months old, weighs in at 110 pounds, and is the friendliest dog you've ever seen.

Lesson #1 is 'heel'. We practice this by using dog treats. Call the dog toward us, holding a treat in the left hand. As the dog approaches, hold the treat out for her and pull it away slowly. Use your fully extended arm to make a circle, ending up with the treat at your side. If the dog follows your hand, she has done a 180 degree turn and should be facing the direction you are facing. Give the treat at this point.

Comment 2 weeks into it: I do lots of morning walks with Sunshine, so I practice this with her a lot. I let her get out ahead of me on her expandable leash. Then I stop and tell her to heal. She was already conditioned to come to me and sit for a treat, so I had to pull her by the leash to get her to heel on my left side. It took several days days, but she got the hang of it. Now she knows the drill. I'm not sure if she will do it automatically unless I have a treat, but she's close.

Week 2

Here is a picture of Sunshine helping us get ready for Halloween. We're putting our decorations in the yard, and I snapped this picture when Nick told her to sit.

We're enhancing our 'heel' command by waiting for the dog to heel (last weeks lesson), then walking a step forward and waiting for the dog to heel up. This isn't hard for me since my hand hangs down exactly where her nose is. She just follows the treat. I'm hoping the Pavlovian-style conditioning is working.

Dealing with Hip Dysplasia

During our advanced training class, we realized that Sunshine was sitting with her back legs more underneath her than other dogs. Our trainer suggested this might be a sign of hip dysplasia. We made an appointment to have Sunshine's hips X-rayed by our vet, and she confirmed the diagnosis–Sunshine's hips are very loose, or dysplastic. But she wanted us to see a specialist who could perform surgery since that might be an option for us. So we contacted the veterinary surgeons in town and took our X-ray images with us.

Considering Hip Surgery

They re-confirmed the diagnosis and recommended a procedure where her hip socket would be rotated downward. This would create a larger surface for the hip bone to sit on and should reduce the symptoms of dysplasia. The surgery is only possible on puppies up to 10 or 11 months old. Once they stop growing, this surgery is not an option. So we had to consider our options quickly.

Of course we want the best for our dog, but at the same time we need some answers about how much benefit the surgery will offer. My meeting with the surgeon was quite frustrating. I wasn't able to get the information I wanted. I was hoping to hear something like ”We see about a 75% recovery rate with dogs in Sunshine's condition”. But the best I got was ”some dogs recover better than others–its all up to nature”. Very frustrating! Neither the analytical nor the emotional sides of my brain were satisfied with that answer.

Cost is also a factor. The surgery runs $2500 per hip, and both of her hips need surgery. Unfortunately, one hip can't be operated on because it has scar tissue around it (the body's natural attempt to grow the hip joint). ”So doctor…you mean we could spend $2500 and still not have the problem resolved?” Yes, thats right. Again…very vague. So we decided to wait and try other options. Surgery is still an option in the future, even when she is an adult dog.

Considering Medication

We put Sunshine on Glucosamine and Chondroitin when our trainer first told us she might have hip dysplasia. She had been on it for 3-4 weeks when we took her to the surgeon. Up to that point, we had seen no benefit from the supplements. Her behavior hadn't changed–she still appeared 'stiff' to us when getting up in the morning. And her energy level hadn't changed.

After a bit of internet research, I discovered a product called Cetyl-M. I googled it, YouTube'd it, and found site after site after site with rave reviews about it. This was very different than the reviews for Glucosamine and Chondroitin. And I even found pet owners with my exact problem who said their dog had incredible results on the medication. So…we gave it a try.

Our package of Cetyl-M arrived in the mail on Friday, Feb 12, 2010. I started giving Sunshine the “loading dose” immediately, which is 2 tablets per 25 pounds of body weight. We saw an improvement in her activity level in 3 days!! By Monday, she was more energetic at home, had more stamina during her walks, and didn't go to sleep right after them.

Two weeks later, she has so much energy that she hops with excitement. She chases things for longer distances (like blowing leaves, squirrels, etc.), and she plays harder with other dogs. I hope these results continue. Best of all, Cetyl-M is fairly cheap–$35 for 120 tablets.

How To Backup an Oracle XE Database

Chris Freyer — Mon, 22 Feb 2010 15:05:54 -0500

How To Backup an Oracle XE Database

There are several kinds of backups available with most database. I'm using an Oracle XE database on one project and have found different kinds of backups to be useful in different situations.

Manual Backup

Manual backups are done in development situations when a single tables needs to be backed up. Multiple tables can be backed up too, but you run the risk of creating an inconsistent backup (because tables are backed up one at a time).

Steps to create a backup:

Login to XE using a browser: http://192.168.1.99:8888/apex. Use the IP/hostname and port number that is valid for your DB.
For each table you wish to backup:
1. navigate to Home –> Utilities –> Data Load/Unload –> Unload –> Unload to Text
2. select the desired database schema, click next
3. select the desired table, click next
4. select the desired column, click next
5. select the desired file export options, click unload data
6. save the file to your machine

My experience with manual backups has taught me to enclose my data in a character that isn't likely to be used in the data, such as the backtick character¹⁾. I also include column names and use a comma as a separator.

Cold Backup

Cold backups are more reliable. but they require the database to be offline. That isn't always possible with production databases. If your uptime requirements allows you to take the database offline for 10-20 minutes, this method will probably work for you.

There are 3 main folders to backup in XE:

the server files (stored in <install dir>/app/oracle/product/10.2.0/server/dbs)
the user files (stored in <install dir>/oradata/XE)
redo log files (stored in <install dir>/app/oracle/flash_recovery_area)

In RedHat Linux, I back them up as follows:

/etc/init.d/oracle-xe stop    #stop the database
cd 
tar cpvfz myxebackup.tar.gz \
     /usr/lib/oracle/xe/app/oracle/product/10.2.0/server/dbs \
     /usr/lib/oracle/xe/oradata/XE \
     /usr/lib/oracle/xe/app/oracle/flash_recovery_area
 
/etc/init.d/oracle-xe start   #start the database

I run this process as root, and it creates a file in my home directory called myxebackup.tar.gz. The backup is complete once the tar command finishes. I can transfer the backup to another machine for safe keeping, or put it in another directory for later restoration.

Hot Backup

Hot backups are performed while the database is running. This can be a labor-intensive task for the database, so expect some performance degradation.

There are two backup scripts included with XE:

/usr/lib/oracle/xe/app/oracle/product/10.2.0/server/config/scripts/backupdatabase.sh
/usr/lib/oracle/xe/app/oracle/product/10.2.0/server/config/scripts/backup.sh

The first shell script simply opens backup.sh in an X11 terminal, so you won't use it unless you are at a graphical workstation. All the real work takes place in backup.sh. It performs a hot or cold backup, depending on whether the database is in Archive log mode or not. To set the database to Archive log mode, follow the steps outlined in the Oracle Database Express Edition, 2-Day DBA guide.

Note that turning on the Archive log means you must take regular backups. The Flash Recovery Area contains a journal of the transactions that have taken place since the last backup. In the event a restore is necessary, the last backup and the flash recovery data can be used to bring the database back to its most recent state²⁾.

Unfortunately, not performing a backup when Archive log mode is on allows the flash recovery area to keep growing, possibly filling up and causing database errors. Hence, a DBA must consider the need for 24/7 operation carefully. It comes at a price, especially considering that Oracle XE can only hold 4GB of data.

¹⁾ on the US keyboard its normally in the top-left corner of the keyboard, to the left of the number “1”

²⁾ think of it like an incremental backup: it holds everything since the last backup

music:william_marshall_s_scottish_melodies

Chris Freyer — Mon, 22 Feb 2010 02:27:06 -0500


X: 1
T:Bog of Gight
C:William Marshall
L:1/8
M:4/4
O:Scotland
Z:Clark Bernst
%%TUNEURL: http://www.leeds.ac.uk/music/Info/RRTuneBk/gettune/00000f04.html
%%ID:00000f04
K:A
c  [| \
e3/2f/ e/c3/2 A3/2c/AE | F3/2E/EA B/B/B Bc | \
e3/2f/ e/c3/2 A/c3/2A3/2E/ | F/A3/2E3/2C/ A,/A,/A, A,2 |
e3/2f/ e/c3/2 A3/2c/A3/2E/ | F3/2E/EA B/B/B B2 | \
cefe dcBA |G3/2A/B3/2c/ A/A/A Ag ||
a3/2b/a3/2e/ a3/2b/a3/2e/ | f3/2e/f3/2a/ b/b/b b2 | \
a3/2b/ a/e3/2 f/a3/2c/a3/2 | B3/2A/ B/c3/2 A/A/A Ag |
a3/2b/a3/2e/ a3/2b/a3/2e/ | f3/2e/f3/2a/ b/b/b b2 | \
a3/2e/ f/a3/2 e/f3/2c/e3/2 | B3/2A/ B/c3/2 A/A/A Ac |]

Welcome

Chris Freyer — Mon, 22 Feb 2010 01:07:04 -0500

Welcome

Welcome to TheFreyers.net. This site has been dedicated to various technical, business, and personal pursuits since 1994. Its structure has evolved from hand-crafted pages (1994-5) to computer-generated HTML (1999) to a forum (2001) to a blogging tool (2005) and most recently to a wiki (mid 2007).

On this wiki, anyone can register. By doing so, you can register for notifications and post comments in discussion areas. If you'd like to edit content, please let me know. I'm happy to give you edit permissions as long as you agree to be a decent human being and stick to the topics at hand.

Common Words

The Most Recent Edits

Page	Date
TECHNOLOGY	2010/03/10 13:30
sidebar	2010/03/10 13:30
How To Use rsync	2010/03/08 15:34
Linux	2010/03/08 14:37
How To Restore an Oracle XE Database	2010/03/03 09:03
Wiki Mindset	2010/03/01 11:43
Adding iLBC to Asterisk	2010/02/25 19:04
The Dog Page	2010/02/25 17:10
How To Backup an Oracle XE Database	2010/02/22 15:05
william marshall s scottish melodies	2010/02/22 02:27
Welcome	2010/02/22 01:07
Moving Recordings to a New Backend	2010/02/21 21:59
Using Windows Authentication In SVN	2010/02/11 12:11
Subversion	2010/02/11 10:59

Moving Recordings to a New Backend

Chris Freyer — Sun, 21 Feb 2010 21:59:01 -0500

Moving Recordings to a New Backend

My current MythTV machine has been expanded as far as it will go, so its time to upgrade. I've built a more powerful server and I want to transition everything to it. I need to move all my data at once to the new server because I don't have enough space to run two servers permanently. And I have 1TB of data, so efficient transport will be an important factor.

Data Transfer

Initially, I attempted to move the TV recordings with a graphical SFTP client. But after several minutes, I realized my transfers were maxing out at about 2.2MBps. That's less than 20% of the capacity of my 100MBps network. I don't have anything else running, so what's limiting my download speed? I decided to investigate. Here are the results of my tests:

Transfer Method	Example Command	Speed
SFTP with hostname	`sftp me@my.machine.com:/video/file.nuv .`	2.2MBps
SCP with hostname	`scp me@my.machine.com:/video/file.nuv .`	2.2MBps
SFTP with IP address	`sftp me@192.168.1.111:/video/file.nuv .`	11.2MBps
SCP with IP address	`scp me@192.168.1.111:/video/file.nuv .`	11.1MBps
SCP with IP & blowfish	`scp -c blowfish me@192.168.1.111:/video/file.nuv .`	11.1MBps
SCP with IP & compression	`scp -C me@192.168.1.111:/video/file.nuv .`	4.4MBps

I read somewhere that using a less CPU-intensive encryption algorithm like blowfish would make my SCP transfers faster, but it didn't help. And turning on SSH compression (the -C) was reported to improve download speeds on slow connections. I tried it on my fast network and it actually made things worse.

So here's what I learned: using a hostname rather than an IP address made a big performance difference. It probably has something to do with my router configuration. I'll ignore that for now…the point is that I got an 11.1MBps transfer rate, which is 91% of the theoretical max of 12.5MBps. The difference is due to protocol overhead.

Data Synch

My MythTV backend is constantly recording new shows. Hence, the set of files I copied to my new server become stale within hours. I need a way to synchronize my new machine with the existing one until I can complete my conversion. Enter rsync.

cd /my/video/folder
rsync -avz me@my.machine.com:/source/folder/ .

I'm forced to enter my password with this command because I specified my user ID, but at least this got my files transferred.

Using Windows Authentication In SVN

Chris Freyer — Thu, 11 Feb 2010 12:11:05 -0500

Using Windows Authentication In SVN

I needed to run an SVN repository on a Windows 2003 Server machine. Installing SVN was pretty simple, but SVN doesn't know anything about Windows. By default, it wants to authenticate against user accounts on the local machine. Getting it to do something different was tricky.

Apache Setup

How do I get SVN to use Windows domain authentication? The short answer is…it won't. The slightly longer answer is…it can, with the help of the Apache web server and some modules. So I installed Apache and started tinkering.

The first step was to edit Apache's httpd.conf file. I needed to enable 4 modules near the top of the file, like this:

LoadModule dav_module modules/mod_dav.so
LoadModule dav_svn_module modules/mod_dav_svn.so
LoadModule authz_svn_module modules/mod_authz_svn.so
LoadModule sspi_auth_module modules/mod_auth_sspi.so

mod_dav is the general-purpose DAV ¹⁾ plugin. mod_dav_svn lets SVN communicate with clients using the DAV protocol. mod_auth_sspi lets clients authenticate against a Windows domain. And mod_authz_svn compares incoming URLs to the directives stored in a configuration file to determine if a user can perform the requested action.

I had to add a new <Location> tag to my httpd.conf file to make Apache aware of my intentions to expose an SVN repository:

  <Location /svn/repos>
    DAV svn
    SVNPath e:\svn\repos
    AuthName "SVN Server"
    AuthType SSPI
    SSPIAuth On
    SSPIAuthoritative On
    SSPIDomain NA
    SSPIOfferBasic on 
    Require valid-user 
    AuthzSVNAccessFile e:\svn\svnaccess.conf
  </Location>

The tag means that access will occur at http://my.server.com/svn/repos, and the repository will exist on disk at e:\svn\repos. Only valid Windows user IDs will be allowed by mod_auth_sspi, and the access rules found in e:\svn\svnaccess.conf will be applied to incoming requests by mod_authz_svn.

SVN Access File Setup

At this point, I could use an SVN client program (like Tortoise, NetBeans, or Eclipse) to reach my repository, but I wasn't being allowed to access anything. I needed to create the access file mentioned in the Apache directive above. The file looks like this:

[groups]
svn-admins = mydomain\cfreyer
developers = mydomain\anne, mydomain\bill, ...
qa_staff = mydomain\anne, mydomain\carl, ...
analysts = mydomain\mike, mydomain\eddie, ...
[release_mgt] = mydomain\jill, mydomain\david

[/]
@svn-admins = rw

[/project1/source]
@developers = rw
@qa_staff = r

[/project1/builds]
@release_mgt = rw
@developers = rw

[/project1/testsscripts]
@analysts = r
@qa_staff = rw

[/project1/reqirements]
@analysts = rw
@qa_staff = r
@developers = r
mydomain\billybob = r

[/project2]
....

The first part of this file is where groups are declared. It begins with the phrase [groups] on a line by itself. Then you create name = value pairs to represent groups. The name can be anything that is meaningful to you–probably a department name or functional responsibility within your organization. The values are a comma-separated list of user IDs. Depending on your setup, you might have case sensitivity issues with user names (I did). Be prepared to debug that by looking at the IDs recorded in the Apache access log.

After the [group] declaration, individual directories (i.e. projects) are declared. Each project is mentioned separately (i.e. [/myproject]). Groups and users are mentioned under the project, along with their access permissions. Its normally better to avoid putting user names inside each project (especially if you have many projects), but its allowed.

Cautions

Here are a few things I've learned the hard way:

The mod_auth_sspi plugin caches user credentials. If a user changes his domain password then tries to access the repository, it won't work. Best solution is to restart apache. Nightly Apache restart will help, but users can be impatient. I haven't found an easy way around this.
I've seen the case of my domain change on different users' machines. In other words, the Apache log will show mydomain\user1 on one day, and MYDOMAIN\user1 on another day. This forces me to duplicate the IDs in the [groups] section.

¹⁾ distributed authoring and versioning

Subversion

Chris Freyer — Thu, 11 Feb 2010 10:59:00 -0500

Subversion

Subversion is a great tool for managing source code. I use it with the Apache mod_dav_svn plugin. The plugin allows SVN to be used with TortoiseSVN, NetBeans, Eclipse, and other client-side tools.

Here is a list of things I've dealt with as it relates to SVN. Note that many of these issues are caused by people monkeying with the server.