I’ve used “wget” for years, but hadn’t bothered to read the man pages for it. It turns out there’s a primo little option you can use to mirror a website, and strangely enough the option is –mirror.

wget --mirror URL

This will essentially get all the html source files, and any files that are linked from within such as images, CSS etc. It even creates the appropriate directory structures so the links all work and the images aren’t broken.

The only thing I’ve found needs doing after this is manually grabbing any background images specified in the CSS files.

Hope this saves you some time too. Props to Helmuth W. Kump for sharing his knowledge.

Table of contents

• Install / Access your base Debian system
• Install and configure the Vim text editor
• Configure your system locale and timezone
• Configure your shell
• Enable public key authentication over SSH
• Create a non-root user for web activities
• Install packages with APT
• Configure Apache Web Server
• MySQL User Management
• Configure PHP
• Character Sets / Encoding and UTF-8
• Hostname, Fully Qualified Domain Name and Reverse DNS
• Configure Exim for sending email

Install / Access your base Debian system

For this guide, we’ll assume your hosting provider has installed a base Debian system and you are accessing your server remotely via SSH as the root user. If you actually have physical access to the machine then you should download a small image file from Debian.org, record it to a CD/DVD/USB disk/floppy, and install using the Internet.

Now is a good time to change the root password..

passwd

Install and configure the Vim text editor

We love Vim (Vi IMproved) for editing all our text based files – let’s install it.

apt-get install vim

If required, uninstall standard vi (package name is nvi) so the “vi” command cranks up vim

dpkg -P nvi

Let’s make a few tweaks to the Vim config file:

vi /etc/vim/vimrc

Uncomment the following line to enable syntax highlighting:

" Vim5 and later versions support syntax highlighting. Uncommenting the next
" line enables syntax highlighting by default.
syntax on

Uncomment the following to have Vim jump to the last position when reopening a file:

" Uncomment the following to have Vim jump to the last position when
" reopening a file
if has("autocmd")
  au BufReadPost * if line("'\"") > 0 && line("'\"") <= line("$") 
    \| exe "normal! g'\"" | endif
endif

Change the default editor for some programs (e.g. crontab, svn) from nano to vim

rm -f /etc/alternatives/editor
ln -s /usr/bin/vi /etc/alternatives/editor

Configure your system locale and timezone

If you receive any locale warnings when running apt commands
e.g. warning: Falling back to the standard locale (“C”).
It probably just means you have to reconfigure your locales with the following command:

dpkg-reconfigure locales

select the appropriate UTF-8 version for your country and set as default
e.g. for us Kiwis en_NZ.UTF-8 is a good choice.

Set correct timezone:

dpkg-reconfigure tzdata

Set the system clock:

date mmddhhmmyyyy

Configure your shell

In the .bashrc file you can specify shell behaviour and appearance, e.g. the way the prompt looks on the command line, being prompted before deleting / overwriting etc.
The following settings are simply personal preference.

vi ~/.bashrc

export PS1='\u@\h:\w\$ '
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'
alias ls='ls --color'
export PROMPT_COMMAND='echo -ne "\033]0;${USER}@${HOSTNAME%%.*}:${PWD/#$HOME/~}\007"'

Enable public key authentication over SSH

Instead of entering a password everytime to SSH into your server, you can enable public key authentication. This can come in quite handy when transferring data between servers or when running scripts where an interactive password prompt would be a pain.
To enable this you need to ensure the following settings are configured in your sshd_config file:

vi /etc/ssh/sshd_config

RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile %h/.ssh/authorized_keys

Restart ssh for changes to take effect

/etc/init.d/ssh restart

Now simply add the public keys of any machines you wish to connect from to the authorized_keys file.

Note: the authorized_keys file must only be writable by user

Create a non-root user for web activities

For good security, we don’t want to run everything as root.
And since most of our activities on this server are to be web based, we’ll create a user called “web”.

adduser web

In general we’ll log into the server as “web” and then switch user to root only when necessary,
e.g. to restart apache etc

Note: bash settings and authorized_keys are per user so you may wish to update these as above for “web”:

su web
vi ~/.bashrc
vi ~/.ssh/authorized_keys
exit

Install packages with APT

OK now we’ve prepared our environment, let’s install a bunch of packages using apt:

apt-get install less
apt-get install mysql-server
apt-get install apache2-mpm-prefork
apt-get install php5
apt-get install php5-mysql
apt-get install php5-gd
apt-get install php5-curl
apt-get install php5-mcrypt
apt-get install php5-cli
apt-get install subversion
apt-get install debian-archive-keyring
apt-get install imagemagick
apt-get install rsync
apt-get install unzip

Configure Apache Web Server

Let’s enable rewrites and SSL in Apache, but turn off the autoindex module (which displays a directory listing to the user when there’s no index file. Run the following commands:

a2dismod autoindex
a2enmod ssl
a2enmod rewrite

If you are enabling SSL, modify the ports config file as follows:

vi /etc/apache2/ports.conf

NameVirtualHost * 
Listen 80 
 
<IfModule mod_ssl.c>
    Listen 443 
</IfModule>

and remove the NameVirtualHost * from the default virtual host

vi /etc/apache2/sites-enabled/000-default

#NameVirtualHost *

To stop warnings on Apache startup, add a ServerName directive as follows:

vi /etc/apache2/apache2.conf

ServerName localhost

Edit the environment variables so Apache runs as our new user “web”:

vi /etc/apache2/envvars

export APACHE_RUN_USER=web
export APACHE_RUN_GROUP=web

MySQL User Management

MySQL comes ready to use, only thing to do is a bit of user management.
By default, MySQL is installed without a password for the root user, so let’s login:

mysql

Note: before running any of the following SQL commands
- be sure to change DB_NAME, DB_USER and DB_PASS first

Firstly, give the root user a password:

SET PASSWORD = PASSWORD('DB_PASS');

Now set up a non-root power user with full privileges to everything:

CREATE USER 'DB_USER'@'localhost' IDENTIFIED BY 'DB_PASS';
GRANT ALL PRIVILEGES ON *.* TO 'DB_USER'@'localhost' WITH GRANT OPTION;

To setup a user who has full access to only 1 specific database:

CREATE USER 'DB_USER'@'localhost' IDENTIFIED BY 'DB_PASS';
GRANT ALL PRIVILEGES ON DB_NAME.* TO 'DB_USER'@'localhost';

Configure PHP

We have installed PHP for Apache and the command line (cli) so we have 2 configuration files to update:

vi /etc/php5/apache2/php.ini
vi /etc/php5/cli/php.ini

We only make a few tweaks here as the default config is pretty good for our needs.
Basically upping the upload filesize and the memory limit, and allowing only cookies for our sessions.

upload_max_filesize = 8M
memory_limit = 128M
session.use_only_cookies = 1

Character Sets / Encoding and UTF-8

Ever seen those nasty question marks showing up on a webpage in place of non-latin characters? Chances are this is due to mismatched or inapporopriate character sets. Something in the pipeline is not set to UTF-8, and it could be mysql, apache, content-type, or even a file encoding.

The best way to avoid these issues is to set things up correctly at the beginning.
Let’s start with MySQL:

vi /etc/mysql/my.cnf

Set MySQL server’s default character set to UTF-8 so all databases, tables and strings will be UTF-8 unless specified otherwise. Add the following line to the [mysqld] section:

[mysqld]
character_set_server=utf8

/etc/init.d/mysql restart

Tip: if for some reason, a database’s default charset is latin1 – then you can issue the following query to display any subsequent utf-8 data correctly:

SET NAMES 'utf8'

Now for Apache.

vi /etc/apache2/apache2.conf

In your your config file, make sure the following directive is present:

AddDefaultCharset utf-8

Include the following HTML meta tag in the head section of your web pages:

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />

Hostname, Fully Qualified Domain Name and Reverse DNS

As an example, let’s say we’re setting up the following server:

          server / hostname = server1
fully qualified domain name = server1.example.com
         primary IP address = 123.456.78.90

Through your DNS admin, point the fully qualified domain name (FQDN) to the primary IP:

$ORIGIN example.com.
server1        A     123.456.78.90     ; ip address for "server1.example.com"

Then through your hosting provider, setup reverse DNS from your IP to your FQDN:

123.456.78.90 - server1.example.com

Finally, in your server’s hosts file point both the FQDN and hostname at the primary IP:

vi /etc/hosts

127.0.0.1       localhost
123.456.78.90   server1.example.com server1

Now on your server you can check things with the hostname command:

hostname -a        # will print your hostname "server1"
hostname -f        # will print your FQDN "server1.example.com"
hostname -d        # will print your domain part "example.com"

Your system’s hostname is set during Debian installation.
You can change the hostname by editing /etc/hostname then running hostname.sh:

vi /etc/hostname
/etc/init.d/hostname.sh start

Note: after changing the hostname, be sure to update your DNS and /etc/hosts as above

Configure Exim for sending email

Exim is a full featured MTA (Mail Transfer Agent) and can be a little overwhelming with all it’s possible configurations. Fortunately we only need it to send email via SMTP for our hosted websites (e.g. sending an order confirmation email from an ecom store), so the config becomes very simple.

By default, mail will be sent from our FQDN (e.g. “server1.example.com” from our example above). You can change that in Exim, but you generally want it to match.

So let’s run the configuration script:

dpkg-reconfigure exim4-config

Answer the questions as follows:

General type of mail configuration:
internet site; mail is sent and received directly using SMTP
 
System mail name:
server1.example.com
 
IP-addresses to listen on for incoming SMTP connections:
127.0.0.1
 
Other destinations for which mail is accepted:
 
Domains to relay mail for:
 
Machines to relay mail for:
 
Keep number of DNS-queries minimal (Dial-on-Demand)?
No
 
Delivery method for local mail:
mbox format in /var/mail/
 
Split configuration into small files?
No

If you have multiple IPs setup on your server, then Exim may choose to send from one of the additional ones as opposed to the primary IP. You can specify which IP to use for sending via SMTP as follows:

vi /etc/exim4/exim4.conf.template

In the “transports” section, add the following line inside the remote_smtp: block:

remote_smtp:
  interface = 123.456.78.90 ; this should be YOUR primary IP

Then restart Exim:

/etc/init.d/exim4 restart

If you’re interested, the config file that gets generated from the template is:
/var/lib/exim4/config.autogenerated