How To: Backup with SSH and Rsync

Posted on by
1

PuTTY logoMy websites (hosted with EvoHosting) are incrementally backed up to a server running on Atlantis – my VMware ESXi host – for the dual purposes of offline development and data resilience. I’m fairly sure that there are numerous guides out there already to accomplish this, but hopefully this will be useful for some.

For ease of explanation, I will skip any network configuration, though if anyone would like detailed configuration on NAT, port forwarding and dynamic DNS, please by all means leave a comment or email me.

SSH Key Configuration

The first thing to do is to configure SSH key authentication, so that your source server can login to your backup server without being prompted for a password. Run the following command on your source server:

ssh-keygen -t rsa

Copy this key into your authorized_keys file (usually in /home/$USER/.ssh) on your backup server and set permissions to 700 so that nobody else can read or edit the file – these permissions are explicitly required for most SSH installations, so double-check that they are correct.

Rsync Configuration

After looking around for help on rsync for a while, this is the rsync command I have come up with:

rsync --delete  -ae 'ssh -p999' /home/craig/public_html/ backup.network.local:/home/craig/public_html/

The “delete” option tells rsync to delete files on the backup that aren’t in the source, and the -p999 option tells SSH to use port 999 for the transfer – I used this as I use a non-standard port for SSH for security, however if you run SSH on its standard port (22), you can remove the -p option.

MySQL Backups

Although Rsync will copy my public_html directory, it won’t backup my MySQL databases. To do this, I used the mysqldump utility. Unfortunately my file permissions wouldn’t allow me to use the more efficient mysqlhotcopy utility, so I have had to make do with a slightly crude full backup, which is gzipped to keep the file size minimal:

mysqldump --all-databases | gzip > ~/databases.sql.gz

This full backup is then imported back into the backup server’s MySQL databases.

Automation via Cron

All that was left to do after this was to schedule the backups automatically via each server’s cron scheduler. The MySQL database imports take place 45 minutes after the Rsync copy to give the backup time to complete.

Where needed, I have written simple Bash scripts to simplify the process and to remove the backup SQL scripts after transfers and imports.

~~

Hopefully that has been educational, please feel free to comment if you have questions/suggestions :)

Multi-booting Windows and Linux using GRUB

Posted on by
2

After my last post on multi-booting different operating systems on the same PC – which was written close to 18 months ago now – I have gained even more experience in setting up multi-boot systems and would like to take the opportunity to revise my previous post and supersede it with this guide.

Partitioning

My only advice with partitioning is to set sensible partition sizes. Most Linux distributions will sit and work happily on 10GB of drive space, though I usually give 20GB to leave some wiggle-room. Don’t forget to incorporate a swap partition for your Linux distribution, the general “rule of thumb” is to use the same amount of space as you have physical memory, so if you have 2GB of RAM, use a 2GB swap partition.

As you will be installing Windows first, you can go ahead and create the partitions for Windows and leave the rest of the space blank.

Installing the Operating Systems

The major difference to the last guide I posted is that this time I will be using GRUB – the Linux bootloader – to choose between operating systems when the machine boots. To do this, first install Windows as normal using your Windows installation media (CD or DVD).

When you have your Windows installation up-and-running, reboot and install Linux. I will be using Fedora as the Linux distribution in this guide as it’s the distribution I am most familiar with, but these instructions should be generic enough to apply to most distributions (Ubuntu, SuSE, Debian, etc).

Proceed with the Linux installation as normal, but look for the screen where the installer configures the bootloader. This could be hidden in an “Advanced” menu somewhere in some distributions, but on Fedora it is the last screen before the installation gets under way.

Your Windows installation should already be listed as “Other”. You can rename this if you want, and you may also be able to change the default option, depending on the options you have within your distribution’s installer. When you have the bootloader configured, move on and install Linux as normal.

When the machine reboots, you should be provided with a screen similar to this:

Congratulations, you now have a working dual-boot system!

Further Configuring GRUB

You can change quite a few options and customise your GRUB screen even further by editing GRUB’s “menu.lst” file. This is found in the /boot/grub folder, and you will have to use root (or sudo) priviledges to edit it as it is a system file.

A lot of in-depth information is available on the GNU GRUB Manual, however here is a quick run-down of a couple of the most useful options:

default – Sets the default option to use. The list is zero-based, so the top option is 0 and increases as you move down the list.
timeout – The number of seconds before GRUB automatically boots using the default option. Set this to 0 to disable the timeout.
hiddenmenu – This tells GRUB to hide the menu and just display a generic “press any key” message. Put a # before this line to comment it out if you want to view the menu by default.

Hopefully the above guide has been useful, please feel free to leave comments if you have any questions or suggestions!

2008 – The Year So Far

Posted on by
1

So, what’s been happening in the last two months?

Linux – Transition Complete

I’ve now finally got Fedora 8 up and running on my desktop PC as well as my laptop so Linux is my main operating system – with Windows being used from time to time for various web design tasks (the recent redesign of my university webspace being one of them) and I have to admit that despite my stance as a self-confessed “Windows user” a year ago, I’m now really enjoying the Linux experience. Everything is so streamlined, but it does take some getting used to and playing around with to get it the way I want it (some of the difficulties and shortcomings being detailed in my transitioning blog) – but I’m generally happy to play around and try different things so maybe this suits me better than others.

SonicaCMS goes live!

SonicaCMS – my plain-text PHP content management system – has now been released as an alpha version (meaning for very early testing and evaluation purposes). It’s been a work-in-progress for quite a while and with a general lull in university work I’ve decided to go public and make the first version available, so if you’d like to download it and try it out, by al means head over to www.sonicadesign.com/scms – I’d be very grateful for any bugs/feedback.

University - New term

As far as university goes, I’m awaiting the results of my first semester exams on Thursday 28th February – some more eagerly than others though. I’ve also actually given a lecture on web development to first years – a somewhat daunting experience but turned out well in the end once I’d gotten over the nerves and into the flow of things. The lecture mainly covered bits and pieces that I’d picked up over the last year and a half of doing websites on a semi-professional freelance basis as well as some search engine optimisation (SEO) material. By the end I actually started enjoying it, so I could be making a return to the lectern at some point in the not-too-distant future…

Along with the new term comes new modules – one of which I’m actually (shock, horror) enjoying: CS35910, aka Internet Services Administration. It generally covers a lot of the things I’ll need to know when I’m out in industry, and it’s material I’m genuinely interested in which always makes a difference. I’ve yet to be given any assignments this semester, but I’m sure they’ll start rolling in soon, as there’s only four weeks left before term ends for Easter.

Liverpool FC - Benitez to go?

Away from computers and all things geek, Liverpool FC are having a bit of a rough period recently, with several high-profile spats between manager Rafael Benitez and the club’s American owners George Gillett and Tom Hicks. After being knocked out of the Carling Cup by Chelsea and the FA Cup today by Barnsley, as well as being effectively ruled out of the Premier League title race by a string of sub-standard performances against sides we should have been beating, we have to yet again concentrate all efforts on the Champions League, which despite all my optimism, isn’t looking too promising either seeing as we’re up against the Italian Serie A champions Inter Milan in the last 16.

I have a feeling that if we don’t progress against Milan, then Benitez’s job will definitely be on the line – more than it ever has done in the past – and, to be honest, I wouldn’t mind seeing him go. Obviously he brought Liverpool the Champions League in Istanbul in 2005 (a night which I will never forget and get goosebumps every time I watch the replays on YouTube) as well as the FA Cup in 2006 in Cardiff and he has secured one of the most important signings in recent LFC history in Fernando Torres, but something tells me that Benitez’s time has come. Maybe I’m being over-critical, but time will tell I think.

We Will Rock You 2008: A New Dawn?

Over Christmas I boosted my show-count to 51, seeing the new cast that took over in September for the first time – and what a cast. You can read detailed reviews about each show on my WWRY website forum, but the shows on 17th December and 10th January certainly gave me the impression that the show is in extremely capable hands, although I haven’t yet seen Sabrina Aloueche in the Scaramouche role – being treated to Sarah French-Ellis on both visits – but I have it on extremely good authority that she is every bit as good as the rest of the cast. My advice – if you’re anywhere near London and are even remotely interested in musical theatre or rock music (or both), book and see this show – you will not regret it.

To go with the new line-up of cast members, an all-new souvenir brochure has bee released, and I have been lucky enough to have been posted a copy by a friend. All fifty-four scans are online on the WWRY-London.co.uk Gallery, I’ve also been busy making my first new banners and wallpapers for over a year, so please check them out! Also in a completely shameless plug, please also feel free to join up on the WWRY-London.co.uk forums.

OK, maybe this entry has been slightly longer than I originally intended, but with two month’s having passed, I thought it was about time for an update.

Multi-Booting: A Revised Guide – XP/Vista/Linux

Posted on by
4

After writing my last quad-boot entry, I’ve gained quite a bit of experience in troubleshooting and sorting out multi-boot installs so I thought I’d re-write my old quad-boot blog to be slightly more like an informative guide rather than a story, and also downgrading it to just cover tri-booting, as I haven’t gotten around to replacing Apple’s OS X Leopard with Sun’s OpenSolaris yet.

Summary

This guide will use Fedora 8 as the Linux distribution, live CDs are available in both Gnome and KDE flavours for those of you who want to try it before actually touching your hard drive’s partition layout. I would recommend installing from the Fedora Install DVD as from experience, the live CD misses out several packages – make and gcc among others – by default that make life a lot easier after installation.

The other operating systems I’ll be dealing with are Windows XP Media Center Edition and Windows Vista Business.

Step One – Partitioning

My desktop PC’s hard drive layout consists of one 200Gb drive for the operating systems and one 320Gb NTFS drive as a central data area, but I’ll assume that you’re dealing with a single drive. Partition whatever space you have into three sections – four if you want a central data area. Although Windows is a lot more space-hungry than Linux, I gave each system an equal amount of space.

You can use whatever tools are available to you to create your partition layout. Both XP and Vista partitions should be NTFS and it’s best to either leave the Linux partition and format this using the install disc. If you’re creating a central data area to use between operating systems, use the FAT32 file system as while Linux can read/write to NTFS, it’s not proven with large file sizes – I use NTFS for my 320Gb data drive purely because FAT32 cannot cope with that size without splitting into separate partitions – something I wanted to avoid.

Step Two – Installing Fedora

Not much instruction is needed at this point. The Fedora Installation Guide is an excellent tutorial through the process. You can install GRUB – the Linux bootloader – to the drive to get the system installed, booting and stable, although booting to Linux will be configured via a GUI utility in Vista later on.

The general guide for creating a swap partition is to use double the amount of RAM you have in your system. My PC has 2Gb of RAM, so I partitioned off 4Gb and set this to be my swap space.

Step Three – Installing Windows

I’ll deal with both Windows systems in the same step here because they’re both relatively similar. You’ll have to do some partition trickery before installation so that Windows assigns the correct drive letters and installs each version in ignorance of anything else on the system.

You can do this via pretty much any partition manager, but I used GParted as included on the Fedora and Ubuntu LiveCDs – or you can use the dedicated 50Mb GParted LiveCD. Select the XP partition, right-click on it and select Manage Flags. Put a tick next to the “hidden” flag and close the window.

Now simply place the XP installation CD in the drive, fire up the installer and install as normal. Just remember to make sure you’re installing to the right partition on your drive. When you’ve installed and can boot to XP, reboot to GParted and swap the “hidden” flags. You can then install Vista as normal.

Step Four – Pulling It All Together

You should now have three working operating systems on your hard drive, but you can only boot to Vista. Head over and download the free EasyBCD program from NeoSmart and install it on Vista. You can then use that to create the necessary boot menu options for each system. EasyBCD includes GRUB and installs it on your Vista partition, so you don’t need GRUB on the Linux partition. Clever eh?

Done!

After you’ve created the boot menu options, reboot into each system once to test. You should now have a working tri-boot system – congratulations! If anything goes wrong or if you’ve got any questions, by all means e-mail me and I’ll do my best to help. If you’re using Fedora as your Linux distribution, head over to the Fedora forums and register.

Hopefully this is more of an informative guide rather than a narrative guide and you’ve found it helpful. If you have (or even if you haven’t), feel free to leave a comment!

Cheers
Craig

Switching to Linux from Windows – Tips from experience

Posted on by
4

In the last few months, I have made the pretty massive jump from being completely Windows-based to being heavily reliant on Linux – the free, open-source alternative operating system. A vast amount of the help and tips I found while in the transition period were found on message forums and blogs, so maybe this will help prospective Linux users decide to switch.

If it ain’t broke, don’t fix it
Don’t switch to Linux “just because”. I had a pretty solid reason to start exploring, but if Windows works for you, then don’t feel pressurised to try Linux. It won’t change your world. It may make your computing experience slightly less stressful, but like 99% of software systems, Linux has bugs – and many of them.

If you’re curious and would like to try it out, get your hands on a LiveCD – there are various Linux distros that can be run from a CD so you can try them before you install, the most popular of which is Ubuntu.

Get used to the command line
Linux has indeed moved on dramatically in recent years to make itself presentable to the (I hate to use the word) “average” user – the person who just wants to surf the web and write letters. That said, while Windows users can often sit in the blissful ignorance that there is such a thing as a command line environment, it’s still very much present in most Linux installations.

Over the six months or so I’ve been running Linux on my laptop, I could not have lived without the command line. Those who have little or no experience with it will either learn or sink. The flip-side of this is that once you have learned a couple of commands, you’ll feel a lot more confident and eventually you will build up experience.

Google is your friend
I also can’t count the number of times I’ve turned to Google with a specific problem and it’s been solved in a matter of seconds with a simple config file edit or command-line instruction. Especially with well-known distros like Ubuntu, support is very easy to find.

Ubuntu is not the only distro out there
Many users refer to Ubuntu in the same was as they refer to Windows – as a complete system. This is completely untrue. Ubuntu is a fairly new entry into the Linux arena and, some say, has only made it to the big-time because of some serious financial clout from a very generous multi-millionaire.

The key to finding a good distro is to research and experiment. You wouldn’t buy a brand new car on just taking a look through a marque list, so why do the same with a computer OS? Ubuntu, for example, didn’t like my laptop’s Intel GMA950 display card, refusing to give me the native 1280×800 resolution, although it detected my Intel 3945ABG wireless card perfectly. Fedora was the exact opposite, but I decided I could work with no wireless for the time being until I found a solution. If in doubt, check the distro’s Hardware Compatibility Lists.

Don’t expect it to work 100% out-of-the-box…
As with the hardware example above, Linux isn’t a “one-size-fits-all” magic system that many Linux fans make out. Personally, my Fedora installation has taken many hours of tweaking and playing with to get it the way I want it – installing media codecs, finding Linux-equivalent software for various tasks, configuring the OS to run the way I want it to.

…But don’t be afraid to play
I haven’t worked with any other distro, but don’t be afraid to get your hands dirty and mess around. If something isn’t how you like it, there’s more than likely a way to alter it. For example, my desktop (screenshot below) has been heavily modified from the standard GNOME desktop layout – the default desktop environment for Fedora and Ubuntu. I’ve made it look the way I want it to look – like Windows – because that’s what I’m used to and I want to use the software because of the functionality, not the look.

Hopefully I’ve given some useful info that’s helped people to decide if they want to switch. I’m quite happily using both Linux and Windows.

My Software
Linux Distro – Fedora 8
Visual Effects Engine – Compiz-Fusion (used to be Beryl on Fedora Core 7)
Window Decorator – Emerald with Crystal-ICE theme

Music – Amarok
Video – VLC & mPlayer
DVDs – Xine
MSN – aMSN (Alvaro’s Messenger)
Email – Mozilla Thunderbird
Web – Mozilla Firefox
FTP – gFTP (standalone) & FireFTP (Firefox extension)
Office suite – OpenOffice
BitTorrent – Azureus
Graphics – Adobe Fireworks CS3 (running under Wine)
Text Editor – Gedit

The Quad-Boot Saga: Background

Posted on by
4

In this post I’ll try and go into as much detail as possible about how I managed to get my Acer 5630 to boot four operating systems. I’ll first give a bit of an overview as to why I chose to install them all. I must point out that I was doing almost all of this flying blind and troubleshooting problems as they arose.

First off I have always been a Windows user since Windows 95 although I’ve really only been seriously playing with it since Windows 2000 and XP. I guess the main motivation for quad-booting was to dispell any opinions that I was a Microsoft “fan-boy” (however much I despise the word) and I was open minded about other systems.

Windows Vista was the first operating system I decided on, as it was what I was using when it was a single-operating system machine. I then stumbled across the gold-mine that is the OSx86 Project and InsanelyMac. For a while I had been using the Apple OS X system for recording, and was fairly comfortable with using it – this made it the ideal choice for a second system.

When I had settled on the idea of dual-booting, I thought about tri-booting as a way of learning more about the UNIX/Linux family – because although OS X is heavily based on UNIX under the bonnet, I still wanted to gain a bit of confidence on the command line and experience Linux as it is. I originally chose Ubuntu as the Linux distro to install, but after trying and failing to get GRUB – the Linux bootloader – to play nicely I chose Fedora Core 7 (the GRUB issue was nothing to do with Ubuntu I later found).

So that left me with three operating systems. After setting them all up and getting them all to play nicely, I thought “why not add a fourth?” – in the typical techie way. It would be nice to have a fall-back Windows system just in case Vista couldn’t handle one program because of incompatability (it wouldn’t be the first time either) – and my Acer actually has a Windows XP Media Center 2005 licence sticker on the bottom not being used, so this became the last piece in the operating system jigsaw.

Next came actually building the thing…..

The Quad-Boot Saga: Stage 1 – Vista & Linux

Posted on by
Reply

After I’d decided which operating systems to install, it was time to get to the dirty work of actually installing them. Seeing as I had dual-booted Windows Vista and Mac OS X before using the Vista bootloader I decided to use this method again for the quad-boot.

A quick run through the major specs of the laptop:
Processor: Intel Core 2 Duo T5500 (2 x 1.66Ghz, 2Mb L2 cache)
Memory: 1Gb DDR2
Hard Drive: Western Digital 120Gb (111.79Gb formatted)

So Vista was installed without any memorable trouble, I gave it a 30Gb NTFS partition because Vista has a reputation for being resource-hungry. Then the fun started – I installed Fedora perfectly onto 15Gb, partitioning it using Fedora’s own installer. I ignored the need for a swap partition because I was going to be short on drive space and I wasn’t planning on using Fedora as the main OS on the laptop.

Getting GRUB to work properly proved to be my first learning curve. Where OS X had simply installed its bootloader on its partition and set it to be active, Fedora installed it onto the bootsector of the hard drive. Through Google I managed to find the commands to enter at the Repair prompt on the Vista installation DVD to re-write Vista’s bootsector:

bootrec /fixboot

bootrec /fixmbr

I then booted to the Ubuntu LiveCD that I had downloaded when I planned on installing Ubuntu. At the terminal, I entered:

sudo grub

find /boot/grub/stage1

I was given the partition number where Linux was installed and therefore where I had to reinstall the GRUB bootloader:

root (hd0,1)

setup (hd0,1)

I then had GRUB setup on the bootsector of the partition and not the drive. I later discovered that this could be done through one of the advanced menus on the Fedora installer. Oh well, live and learn :) .

Entering quit then returned me to the main command prompt, where I then had to take a disk-dump of the bootsector so I could feed it into the Vista bootloader:

sudo fdisk -l

sudo dd if=/dev/sda2 of=/linux.bin bs=512 count=1

The first command lists all of the partitions on the disk so I can see how Linux labelled them. The second command actually took the first 512 bytes (i.e. the first sector) of the partition and dumped it inside a file on the root drive called “linux.bin”. I then transferred this to my USB memory stick to put onto Vista.

Once I had my copy of GRUB dumped into a file, I booted into Vista and fired a command prompt. The “boot.ini” method has been changed to a command-line based bootloader, so if you’re confident with the command line you’re ok. If not, download EasyBCD from NeoSmart. Assuming you have called the file “linux.bin” and it’s in the root C:\ drive:

bcdedit /create /d “Fedora Core 7 – Linux” /application BOOTSECTOR

That will give you a long identifier string to use with the rest of these commands – in place of {YOUR-ID-HERE}:

bcdedit /set {YOUR-ID-HERE} device partition=C:

bcdedit /set {YOUR-ID-HERE} path \linux.bin

bcdedit /displayorder {YOUR-ID-HERE} /addlast

After that, I rebooted and I had a lovely dual-boot setup with Linux and Vista :D .

The Quad-Boot Saga: Stage 2 – OS X and Media Center 2005

Posted on by
Reply

This was the slightly easier bit. Once I had Vista and Fedora happily coexisting, I added OS X. Before I continue, I would like to point out that I am writing this for educational purposes only and in no way do I take responsibility for any actions taken by visitors to this site :) .

With that little bit out of the way,I had installed and re-installed OS X more times than I can care to remember during the dual-boot phase with the same laptop, trying various different kexts and solutions to get things like the audio and ethernet working. I used this guide to install OS X and used the JaS 10.4.8 install disc.

OS X installation went as smoothly as it could, I did experince some kernel panics during boot at one point, I’m not sure exactly what was wrong but a reinstall solved the problems. After following the instructions in the guide to add OS X to the Vista boot menu, I had a fully functioning tri-boot system :) .

This is where the major problem occurred. Due to the fact that the system I was attempting to quad-boot was a laptop, it only had one hard drive. I found that the partition table of any hard drive can only have four entries – meaning that my initial plan of having one partition per operating system and one FAT32 partition for music and documents (FAT32 is the only filesystem that can be reliably used by all three platforms) wasn’t going to work – unless I could convert one or more of the current partitions into a logical volume.

With quite a bit of research on Google and various message forums, I converted both OS X and Fedora to logical partitions using Acronis Disk Director on Hiren’s BootCD – that CD has been an absolute God-send for me, the amount of utilities that are included with it is astounding. After converting the partitions and creating the last two to be used for XP and data storage, a had a configuration which looked like this:

Vista – 30Gb NTFS (primary)
Fedora – 18Gb ext3 (logical)
OS X – 26Gb Mac OS Journalled (logical)
Media Center - 8Gb NTFS (primary)
Data Storage - 28Gb FAT32 (primary)

I know this isn’t the ideal configuration (three logical partitions sandwiched between two primary) but as I said in the introduction post, I was doing this as I was going along and using it as a learning experience :P .

Then came the Media Center install, that went pretty smoothly as well (to be honest the only OS that didn’t was Ubuntu – and that was GRUB’s fault), although before I ran the install I had to do a bit of partition hiding using GParted on the Ubuntu LiveCD. This is because without hiding the other NTFS and FAT32 partitions on the drive, the Media Center install will automatically copy the related boot files to the first readable partition it finds on the drive – and also set drive letters accordingly.

So I booted to the Ubuntu LiveCD, fired up GParted and set the “hidden” flags on the Vista and Data Storage partitions, and also set the Media Center partition as “boot”. After that, I ran the Media Center install without a problem, and booted into it a few times to check that all was well. I then booted back into GParted, removed the “hidden” flags and set the Vista partition as “boot” again.
The last thing left to do was add the entry into Vista’s bootloader. I had to copy three files to the C:\ drive from the XP installation drive – ntdlr, boot.ini and ntdetect.com. Once these were all in place, I created a new Legacy Bootloader using EasyBCD – I was getting lazy at this point – and rebooted with fingers crossed.

And that’s about it. It has been one hell of learning experience for me, and I hope that this series of blogs are of use to some people out there.

Thanks for reading!