During a recent upgrade of VMWare ESXi from 4.1 to 5.5, I encountered a problem: after the upgrade, systems stored on a shared iSCSI storage platform did not work. In fact, a number of them were damaged. Files could be read and downloaded from the datastore with no issues, but writes to the datastore failed.

A number of symptoms appeared, most tellingly in the VMWare client console about connectivity to the datastore being lost and then restored. However, the vmkernel logs on the hosts themselves showed more interesting failures, specifically reservations and the failure of SCSI ‘WRITE_SAME’ 0x93 errors. This led me to a post about the Linux SCST target software, stating that this doesn’t support this command and to disable VAAI/acceleration features in VMWare.

This did solve the problem and it appears that VMWare has been a bit dumb here. The options are marked as ‘requires compliant hardware’ inside VMWare but 1) they are turned on by default, and 2) are applied to targets with a HW acceleration value of ‘Unknown’, rather than being restricted to validated hardware. Poor design decision in my opinion.

To turn off these features, you need to edit the following on each host:

ESXi Configuration > Advanced Settings

/VMFS3/HardwareAcceleratedLocking > Set to 0

/DataMover/HardwareAcceleratedMove > Set to 0

/DataMover/HardwareAcceleratedInit > Set to 0

 

(Originally posted 2015-06-24)

So, I ran into something interesting and confusing today.

I found that my PDC was stuck on ‘Local CMOS Clock’ as it’s source, despite it being configured to use an external NTP server (in this case, my local Cisco router is configured as an NTP server). Finding this odd, I verified the configuration and then executed a resync command:
w32tm /resync
and received this output:
Sending resync command to local computer
The computer did not resync because no time data was available.
Curious, as the NTP server is running and other devices are syncing correctly. Logging into the Cisco router and executing ‘show ntp associations’ resulted in this:
 address         ref clock       st   when   poll reach  delay  offset   disp
  192.168.1.11    .INIT.          16      –  32768     0  0.000   0.000 15937.
  192.168.1.12    .INIT.          16      –  32768     0  0.000   0.000 15937.
 ~2610:20:6F15:15::27
                  .STEP.          16      –   1024     0  0.000   0.000 15937.
*~128.138.140.44  .NIST.           1      2     64   377 44.098   0.552  3.122
Well, that’s odd. The system is syncing correctly, but why are the domain controllers showing up as stratum 16?
‘show ntp associations detail’
192.168.1.11 dynamic, insane, invalid, unsynced, stratum 16
ref ID .INIT., time 00000000.00000000 (16:00:00.000 PST Wed Dec 31 1899)
our mode passive, peer mode unspec, our poll intvl 32768, peer poll intvl 131072
root delay 0.00 msec, root disp 0.00, reach 0, sync dist 15956.69
delay 0.00 msec, offset 0.0000 msec, dispersion 15937.50
precision 2**24, version 3
Well, that’s not good, it’s trying to register itself as an NTP peer with the Cisco router. Googling some bits finally led me to a bug report between certain versions of Cisco IOS and W32Time.
The fix?
Change the w32time configuration:
w32tm /config /manualpeerlist:IPADDRESS,0x8 /syncfromflags:manual /reliable:yes /update
net stop w32time && net start w32time
w32tm /resync
That ‘0x8’ allows it to work.

(Originally posted 2013-05-29)

Man, I’m so tired of dealing with scamware… fortunately I don’t have to do it much, but when your mother calls…

So the latest variant of this Internet Security malware is out and infecting people, and there are already guides on how to fix it. Most of them are fine, use Malwarebytes, re-run your AV afterwards etc… the problem is, if you are running WinXP, it plays a nasty trick against your AV software: it changes the files to junction points. In Vista+, this isn’t a problem, since the system can handle it. XP can’t, and also sees the junctions as corrupted. I’ve seen this with MS Security Essentials so far.

There are two ways to fix this: 1) Slave the drive to another computer or use a Linux-based rescue CD to remove the junctions.

Or, if you are running XPSP3 (which you should be!) you can use the ‘fsutil’ tool to delete the junction point and restore the file.

If you try and reinstall MS Security Essentials without fixing the junctions, you’ll get odd error codes.

To remove the junctions:

Command window (cmd)

Switch to the directory containing the junctions ( C:\Program Files\Microsoft Security Client\  for example)

Dir /p

You should see a list of files/directories with the type <JUNCTION>.

For each file, run ‘fsutil reparsepoint delete filename

After that is done, you can remove and reinstall the software.

(Originally posted 2012-11-13)

DISCLAIMER: I take no responsibility for any disasters that occur due to your use or non-use of these commands. You are supposed to ask Equallogic support for permission 😉

support max-repl-segments # – this sets how many segments of data will be “in-flight” at a time

support repl-use-jumbo YES/NO – enable/disable jumbo frame use for replication traffic ONLY

support max-repl-xfer # – how many simultaneous replications will run at a time; replications beyond this number will wait for an existing transfer to complete in full before starting

support repl-window-size #KB – TCP window size in kilobytes; tune this according to your line speed, latency and how much bandwidth you want to utilize per replication.

(Originally posted 2012-06-16)

Based on ASDM 6.4(7), and ASA OS 8.2(5).

If you want to use your AD accounts for Administrative logins to the ASA, as well as Execute access, this will help you configure it. You’ll need to configure a group in AD and add the relevant accounts that you wish to grant access to beforehand.

ASA AAA for Administration

Expand: ASDM > Configuration > Device Management > Users/AAA

Select AAA Server Groups
Select ADD under ‘AAA Server Groups‘ pane
AAA Server Group: LDAP_DOMAIN
Protocol: LDAP
Reactivation: Depletion
Dead Time: 10 minutes
Max Failed Attempts: 3
Ok

Highlight ‘LDAP_DOMAIN‘ in the list

Select ADD under ‘Servers in the Selected Group‘ pane
Interface Name: Select Interface which the LDAP server is accessible on
Server Name or IP Address: IP of LDAP server
Timeout: 10 seconds
Enable LDAP over SSL: uncheck  (if you have a proper PKI certificate configured, you can enable this)
Server Port: 389   (port 636 by default for SSL, if you have enabled it)
Server Type: Microsoft
Base DN: Base bind point of the directory search in CN=XXX,OU=XXX,DC=XXX,DC=XXX format
Scope: Sub
Naming Attributes: sAMAccountName
Login DN: CN=LDAP Reader,OU=XXXX,DC=XXXX,DC=XXXX  (just a domain user, not a domain admin)
Login Password: password of LDAP Reader account
LDAP Attribute Map: –None–  (note, we’ll come back to this setting)
SASL MD5 Authentication: uncheck
SASL Kerberos Authentication: uncheck
Group Base DN: leave blank
Group Search Timeout: 10
OK

Expand ‘LDAP Attribute Map‘ at the bottom

Select ADD
Name: LDAP_MemberOf_ServiceType
Under Mapping of Attribute Name tab:
LDAP Attribute Name: memberOf
Cisco Attribute Name: IETF-Radius-Service-Type
Select ‘Add >>
Select ‘Mapping of Attribute Value’ tab
LDAP Attribute Value: group name in CN format to valid users
Cisco Attribute Value: 6
Select ‘Add >>
OK

Select Server added under ‘Servers in the Selected Group

Select Edit
Change LDAP Attribute Map to ‘LDAP_MemberOf_ServiceType
OK

Highlight the Server that was just added in the list

Select Test
Select Authorization
Enter user that should be authorized successfully
Ok
Select Test
Select Authorization
Enter user that should NOT be authorized successfully
Ok

You can repeat the above steps, except for the LDAP Attribute Map, which can be reused, for multiple servers for redundancy. I highly recommend this.

Select ‘User Accounts‘ in navigation pane

Select ‘Add
Username: some_admin_user
Password: some_admin_password
Confirm Password: some_admin_password
Select ‘Full Access
Change Privilege Level to ‘15
OK

**This is a LOCAL database user that will be used for emergency access to the ASA if the LDAP servers are unavailable. This account cannot log in if the LDAP servers are available.**

Select AAA Access in navigation pane

Select ‘Authentication‘ tab
Under ‘Require authentication to allow use of privileged mode commands
Check ‘Enable‘, Server Group: LDAP_DOMAIN, check ‘Use LOCAL when server group fails
Under ‘Require authentication for the following types of connections
Check ‘SSH‘, Server Group: LDAP_DOMAIN, check ‘Use LOCAL when server group fails
Leave HTTP/ASDM, Serial and Telnet unchecked, this way if you’ve screwed up somewhere, you haven’t locked yourself out of the ASA.
Select ‘Authorization‘ tab
Under ‘Enable authorization for ASA command access
Uncheck ‘Enable
Under ‘Perform authorization for exec shell access
Check ‘Enable‘, ‘Remote server
OK

Apply configuration

Test connecting to the ASA via SSH via domain credentials; no need to specify domain name with the username. Verify that executing ‘enable’ and specifying your AD password works.

If successful, save configuration to flash. Otherwise, review configuration via ASDM and change/test until it does.

Once satisfied that access is correctly being handled via SSH, enable LDAP AAA for HTTP/ASDM as well.

Select AAA Access in navigation pane
Select ‘Authentication‘ tab
Under ‘Require authentication for the following types of connections
Check ‘HTTP/ASDM‘, Server Group: LDAP_DOMAIN, check ‘Use LOCAL when server group fails
OK

Save configuration to flash/startup-configuration.

(Originally posted 2011-06-16)

For Windows clients to automatically find a KMS server, it needs to have a SRV record in DNS. This should be located as follows:

DNS Server -> Domain Name -> Other New Records

Select Service Location (SRV)

  • Service: _vlmcs
  • Protocol: _tcp
  • Priority: 0
  • Weight: 0
  • Port number: 1688
  • Host offering the service: FQDN of KMS Server

Once you have this, you can either wait for the next time Product Activation runs in the background, or you can force it by running the two following commands via an elevated command shell:

  • cscript %windir%\system32\slmgr.vbs -skms FQDN.OF.KMS.SERVER
  • cscript %windir%\system32\slmgr.vbs -ato

This will run activation against the specified KMS server.

(Originally published 2010-08-19)

Configuring ASA 8.2 for Remote VPN Access, with LDAP Authentication and Authorization

The following components need to be configured:
– AAA Server
– LDAP Attribute Map
– Address Pools
– Group Policy
– Connection Profile/Tunnel Group

This guide assumes you are using ASDM 6.2 for configuration of the ASA.

This guide was written and tested against a Cisco ASA 5520, Firmware version 8.2(2), ASDM version 6.2(5)

LDAP Attribute Map Configuration

– Create a normal Domain User account for the ASA to use to bind to the Directory to query. This user needs no special access other than to be able to query the directory. It certainly does not need to be a domain administrator account!
– Create an AD Group to be used for access to the VPN. Name it something sensible.

– Get the DN’s for the user and group created, you’ll need them for the next steps. You can use dsquery or the Object tab in ADUC to get them.

– On the ASA, go to Configuration > Remote Access VPN > AAA/Local Users > LDAP Attribute Map
– Create a new map, call it something like ‘LDAP_memberOf
– For the Mapping of Attribute Name:
LDAP Attribute Name: ‘memberOf
Cisco Attribute Name: ‘Group-Policy
*NOTE: Below ASA 8.2, this is IETF-Radius-Class, it has been renamed for 8.2+
– For the Mapping of Attribute Value:
LDAP Attribute Value: full DN of the Remote Access Group from AD e.g. ‘CN=RemoteGroup,OU=Groups,DC=domain,DC=com
Cisco Attribute Value: The name of the ASA VPN Group Policy you are going to use (we haven’t created it yet) e.g. ‘RemoteAccess_Grp
-Save

AAA Server Configuration

– Go to Configuration > Remote Access VPN > AAA/Local Users > AAA Server Groups
– Add a new AAA Server Group:
Server Group: Descriptive Name i.e. ‘LDAP_DOMAIN
Protocol: LDAP
Reactivation Mode: Depletion
Dead Time: 10 minutes
Max Failed Attempts: 3
– OK
– Now, with the LDAP_DOMAIN Server Group selected in the top frame, click on Add for the ‘Servers in the Selected Group’ frame
Interface Name: INSIDE or whatever interface you have your Domain Controller behind
Server Name or IP Address: IP Address of a Domain Controller
Timeout: 10 seconds
Server Port: 389 (unless you enable the LDAP over SSL option)
Server Type: Microsoft
Base DN: ‘DC=domain,DC=com‘ or whatever your base DN is
Scope: ‘All levels beneath the Base DN
*Note: One level is quicker, but you need to set the base DN to where your AD Access Group is located
Naming Attribute(s): ‘sAMAccountName
Login DN: full DN of the normal AD User created above
Login Password: password of the AD User account
LDAP Attribute Map: ‘LDAP_memberOf‘ or whatever you called the LDAP Attribute Map previously created
SASL options unchecked
Group Base DN: leave this blank
– OK

Address Pools

– You can either use a DHCP server, or assign a static pool to the ASA to hand out to VPN clients.
– For DHCP: Configuration > Remote Access VPN > Network (Client) Access > Address Assignment > Assignment Policy:
– Uncheck ‘Use Authentication Server
– Check ‘Use DHCP
– Uncheck ‘Use Internal Address Pools
– For Static Pool (which I recommend): Configuration > Remote Access VPN > Network (Client) Access > Address Assignment > Assignment Policy:
– Uncheck ‘Use Authentication Server
– Uncheck ‘Use DHCP
– Check ‘Use Internal Address Pools
– Uncheck ‘Allow the reuse of an IP address XX minutes after it is released
Configuration > Remote Access VPN > Network (Client) Access > Address Assignment > Address Pools
– Add New Pool
Name: Descriptive Name e.g. ‘RemoteVPN_Pool’
Starting Address: IP
Ending Address: IP (note these are inclusive!)
Subnet Mask: Mask
– Ok

*Note: The rest of this guide assumes a Static Address pool was created.

Address Pool created

Group Policies (Note, the terminology sucks. This is a policy for a group on the ASA, not an AD GPO)

– Go to Configuration > Remote Access VPN > Network (Client) Access > Group Policies
– Create a new policy to deny access:
– Add Internal Group Policy
– General:
Name: NoAccess
Address Pools: uncheck ‘Inherit‘ and leave blank
Expand ‘More Options’:
Simultaneous Logins: uncheck ‘Inherit‘ and set to 0
– Ok

This policy will be used to deny unauthorized & unauthenticated users from connecting.

– Create a new policy to for authenticated users:
– Add Internal Group Policy
– General:
Name: ‘RemoteAccess_Grp
Address Pools: uncheck ‘Inherit‘ and select the Address Pool previously created
Simultaneous Logins: Uncheck ‘Inherit’ and set the value to the number of Remote users you have licenses for or the number of valid IP addresses in your VPN pool
– Servers:
DNS Servers: I generally explicitly set them. Delimit with comma or space
WINS Servers: If in use, I generally explicitly set them. Delimit with comma or space
Expand More Options:
Default Domain: uncheck ‘Inherit‘ and set explicitly
– Advanced:
Split Tunneling: If you want to enable split tunneling, set as below. The default is tunnel all.
DNS Names: ‘Inherit
Policy: uncheck ‘Inherit‘ and select ‘Tunnel Network List Below
Network List: uncheck ‘Inherit‘, click ‘Manage
Create a new ACL with permit ACEs that include the hosts/netblocks you WANT tunneled.
Other options: Leave rest at default of ‘Inherit
– Ok

Group Policies created

Connection Profiles / Tunnel Groups:

Go to: Configuration > Remote Access VPN > Network (Client) Access > IPsec Connection Profiles
Click on Add
Name: Descriptive name e.g. ‘RemoteAccess_TunnelGroup’
IKE Peer Authentication:
Pre-shared Key: enter IKE Pre-shared key here
Identity Certificate: None
User Authentication:
Server Group: ‘LDAP_DOMAIN
Fallback: ‘Use LOCAL if Server Group fails‘, check this if you want to fall back to local users if AD is unavailable
Client Address Assignment:
DHCP Servers: leave blank
Client Address Pools: Select Address Pool previously created
Default Group Policy:
Group Policy: NoAccess
*Important, the default policy should be to lock them out, successful authentication&authorization provides the access policy
Advanced
Authorization:
Server Group: ‘LDAP_Domain
Users must exist in the authorization database to connect: Check this
-Ok

Enabling access to VPN clients:

To enable IPsec access:
– Go to Configuration > Remote Access VPN > Network (Client) Access > IPsec Connection Profiles
– Under Access Interfaces, check ‘Allow Access‘ for your ‘OUTSIDE‘ interface

To enable AnyConnect SSL / Legacy SSL:
– Go to Configuration > Remote Access VPN > Network (Client) Access > AnyConnect Connection Profiles
– Under Access Interfaces, check ‘Allow Access‘ for your ‘OUTSIDE‘ interface
– Under Connection Profiles, edit the tunnel group you created and give it an alias (The end users will see this in AnyConnect)
Login Page Setting: check ‘Allow user to select connection profile….
– SSL VPN clients also require a SSL certificate to be installed on the interface:
Configuration > Remote Access VPN > Advanced > SSL Settings
Edit your ‘outside’ interface and either create a self-signed certificate (warning will show on the client, but will work) or provide an existing certificate

NAT Rules
Configuration > Firewall > NAT Rules
If ‘Enable traffic through the firewall without address translation‘ is checked, you don’t need to add anything here
Otherwise:
Create a new NAT Exempt Rule:
Original
Interface: INSIDE
Source: Network object containing tunneled networks (generally your internal netblock, but you can use this to restrict devices VPN users can get to)
Destination: Create a new network object containing your VPN Client addresses
NAT Exempt Direction:
NAT Exempt outbound traffic from interface ‘INSIDE’ to lower security interfaces(Default)
Description:
Write a handy reminder for yourself when you are staring at this rule 6 months from now during a security audit.
– Ok

NAT Rules done

That should be everything, you should have IPsec and SSL VPN capability to your ASA now. Install the appropriate client on the end device and test.

Clients:

Both the Anyconnect SSL client and the IPsec VPN client are available for a variety of operating systems, and can be pre-installed and configured for the end-user.

On the IPsec VPN client, the Group name and Password are the Tunnel-Group name and the IKE pre-shared key. It will prompt for the user credentials afterwards.
PCF files can be created to securely distribute the IKE PSK to remote end-users (saved as a one way hash in the PCF file).

(Note this was originally published 2010-04-05 and has not been updated)

Pulling your Exchange valid addresses to a Postfix server

WARNING: Make sure you understand the security consequences of creating a passwordless/passphraseless account on your system. Appropriate security measures should be taken to prevent systems from being compromised.

You’ll need to decide where to set up this function. You can do it directly on the receiving Postfix server, but since it’s probably in a DMZ, that would mean allowing access from your DMZ back into your protected LAN with AD. It’s generally a bad idea to open access from the DMZ to the LAN (obviously, there are times where this is unavoidable). Instead, what I recommend is you have a system on your LAN that will push the information up to the DMZ host. This is more secure, since the connection is initiated from your LAN.

The general method is as follows:

  1. Management box polls AD servers for addresses and builds a file
  2. Management box then sends the file to Postfix server
  3. Management box initiates remote update of the Postfix server mapping

What you need on the Management box: You’ll need a working install of Perl, with the Net::LDAP modules installed. You’ll also need this script, which polls the LDAP information and builds the file for sending to Postfix.

First off, grab that script. You’ll need to edit the settings at the beginning to fit your AD domain. You’ll need an AD account that can read AD information. DO NOT USE A DOMAIN ADMIN ACCOUNT! You should use an account with privileges to only access the domain. The password for the account will be in the file as plain-text, so choose accordingly. You can also change the name and location of the resultant file the script creates; I’ve left in my default of /scripts/exchange_recipients

Once the script is appropriately modified, upload it to the management box and place it wherever you like to keep scripts. I generally make a /scripts directory and put it there. I’ll assume for the remainder of the tutorial you’ve done the same. You’ll need to make sure it’s executable as well, so run ‘chmod 664 /scripts/getadsmtp.pl‘ on it.

Now, let’s create our automation account. First on the management server, add a new user, with a blank password. For the guide, lets refer to this account as autoscript. Once the account is created, you’ll need to change to it ‘su autoscript‘ and create an SSH key for it. First change to the home directory ‘cd ~‘ and create a new directory, ‘mkdir .ssh‘. Set permissions: ‘chmod 700 .ssh‘. Change to: ‘cd .ssh‘. Create an RSA SSH keypair by running ‘ssh-keygen -t rsa‘. Answer the questions, leaving the passphrase blank. You should have two files in your .ssh directory now, one is the private key (id_rsa by default) and the other is the public key (id_rsa.pub by default).

Now, on the Postfix server, you’ll need to create the same account and .ssh directory. Once you’ve done this, copy the id_rsa.pub file from the Management server, to the Postfix server. Then, from wherever you have stored the public key, you’ll want to run the following commands:

  1. cat id_rsa.pub >> /home/autoscript/.ssh/authorized_keys
  2. cat id_rsa.pub >> /home/autoscript/.ssh/authorized_keys2
  3. chmod 664 /home/autoscript/.ssh/authorized_keys
  4. chmod 664 /home/autoscript/.ssh/authorized_keys2

While we are on the Postfix server, we need to do some other setup. First off, lets make it so that the autoscript account can run certain commands as root without providing a password. To do this, we’ll need to edit the sudo control file: ‘visudo‘. Add the following line at the bottom of the file: ‘%autoscript ALL=NOPASSWD: /usr/sbin/postmap‘. This allows autoscript to run postmapwithout providing a password.

We also need to set up some files in the postfix directory to receive our Exchange address file. Do the following (assuming postfix is in /etc/postfix):

  1. cd /etc/postfix
  2. touch exchange_recipients (assuming default name of file)
  3. touch exchange_recipients.db
  4. chown root:autoscript exchange_recipients
  5. chown root:autoscript exchange_recipients.db
  6. chmod 664 exchange_recipients
  7. chmod 664 exchange_recipients.db

This precreates the files that will be updated by our script with the proper permissions. If you try and write these files remotely, it will fail because autoscript doesn’t have privileges to create files in the /etc/postfix directory.

Almost done!

TEST THE getadsmtp.pl SCRIPT!– Make sure the script is creating a valid file and placing it where you want it. Once you’ve verified this works, continue on.

On the management server, you’ll need to create a script to run getadsmtp.pl and send the results to the Postfix server. Lets do that now:

vi /scripts/update_edge_recipients.sh

Copy in the following code, updating $REMOTESERVER for the IP or hostname of the Postfix server (I recommend IP for this):

#!/bin/bash

#Transfer and Update Script for Postfix
#Requires getadsmtp.pl and exchange_recipients file to be created and accessible by autoscript user
#Remote server needs to be prepared before update script will work: remote exchange_recipients and .db file need
#to be writeable by autoscript user

/scripts/getadsmtp.pl
scp /scripts/exchange_recipients autoscript@$REMOTESERVER:/etc/postfix/exchange_recipients
ssh autoscript@$REMOTESERVER sudo postmap/etc/postfix/exchange_recipients

# EOF

Now you’ll need to set permissions and ownership of the script files so they can be executed.

  1. chown autoscript:autoscript /scripts/update_edge_recipient.sh
  2. chmod 755 /scripts/update_edge_recipients.sh‘.
  3. chown root:autoscript /scripts/getadsmtp.pl
  4. chown autoscript:autoscript /scripts/exchange_recipients
  5. chmod 774 /scripts/exchange_recipients

That should set ownership and permissions correctly. Switch to the autoscript user and run ‘/scripts/update_edge_recipients.sh‘. Look for any errors. If it asks you for a password, make sure you didn’t misspell either of the authorized_keys files on either server. Otherwise, you should just see some quick text about copying the file.

Two things left: Set the Postfix server to use this file, and add this job to the cron to run automatically.

On the Postfix server, edit main.cf and add another or edit your relay_recipient_maps to use the new file like this: relay_recipient_maps = /etc/postfix/exchange_recipients. Make sure to reload Postfix to use the new configuration.

On the management server, you’ll need to edit the cron for the autoscript user (NOT root’s cron!). Do this by changing the the autoscript user and running ‘crontab -e‘. Then add in the following line to the crontab: ‘0 * * * * /scripts/update_edge_recipients.sh‘ This runs the script at the start of every hour, every day. You can adjust the timing to fit your needs.

Congratulations, you’re done!

(System commands based on Ubuntu 8.04. Thanks to Chris Covington for writing the original getadsmtp.pl script.)

(Note this guide was originally published 2010-02-11 and has not been updated)

This article was developed using an Ubuntu 8.10 box, but since it uses mostly compiled from source applications, it should be easily portable to the distribution of your choice.

So, you want to offer some web services, but your company is standardized on MS-SQL for their backend? I can already hear the Linux die-hards yelling about Postgres and MySQL already, but fact of life is, a lot of companies use MS-SQL. But there’s no reason to not use a nice lightweight, fast, open-source front end for your web apps against this database.

Three major applications we’ll use to provide access in this guide are: FreeTDS, an interface application that allows direct SQL access, Apache 2.2 for web server duties, and PHP 5.2.6 for scripting/DB access/dynamic HTML generation.

First off, you’ll need to load up your Linux box with the regular tools used for building applications (gcc, etc). This can be done under Ubuntu by getting the ‘build-essential’ package.

Now, in your home directory, you’ll want to pull down the source packages for the applications we are going to build. You can pull down Apache and PHP source packages in Ubuntu with ‘apt-get source packagename‘. For FreeTDS, you can get the latest package from the FreeTDS website (link here). Use either FTP or wget to download the tar.gz file and extract it.

First off, lets build Apache. It’s a relatively simple configure, since we really only need the dynamic module loading capability compiled in. Switch the the source directory for Apache, and execute the following configure script:

./configure –prefix=/usr/local/apache2_2 –enable-so

You’ll see a bunch of text fly by. Congratulations, you’re compiling code! Once that is done, it will have created what is called the ‘makefile’. Guess which command you run next… yup.

make

More text flies by! Once it’s done, run ‘make install‘. This will put the compiled libraries in their spots.

Alright, so Apache is now installed under /usr/local/apache2_2.

Now lets build and install FreeTDS. This needs to be built before we can compile PHP.

Change to the extracted source directory of FreeTDS (you did unpack the tar.gz archive right?). Again, we’ll go thru the same steps as Apache for building.

./configure –prefix=/usr/local/freetds  –disable-odbc
make
make install

In this setup, we’ve disabled building the ODBC driver for FreeTDS, as we are not using a Linux ODBC driver manager, so we don’t need it. We use FreeTDS as a direct interface to MS-SQL.

Now we can compile PHP. I used version 5.2.6 and we’ll be enabling some common features used by PHP scripts. Switch to the source directory for PHP and execute:

./configure –prefix=/usr/local/php5 –with-apxs2=/usr/local/apache2_2/bin/apxs –with-gd –with-mssql=/usr/local/freetds –enable-calendar

What this does is sets the build location to /usr/local/php, the ‘with-apxs2‘ is the apache module support for dynamic loading, ‘–with-gd‘ enables graphic support, ‘–with-mssql‘ enables mssql functions within PHP, and ‘–enable-calendar‘ builds the calendar manipulation features.

Note: the ‘–with-gd‘ option usually requires PNG support. You’ll need the development libraries for it. Under Ubunutu, you can get these with ‘apt-get install libpng12-dev‘.

Once you’ve got a successful configure, execute the make and make install commands.

Congratulations, you’re almost done.

Check the /usr/local/apache2_2/conf/httpd.conf and make sure there is a line in the LoadModule section as follows:

LoadModule php5_module    modules/libphp5.so

This enables PHP5 support for Apache. You will need to restart Apache if you modify the httpd.conf (or recompile PHP for additional options later).

You’ll also need to edit the freetds.conf file to put in your SQL server settings. This is a dynamic file that is checked on each call, so you don’t need to recompile or restart anything if you make changes. The TDS version line value is generally set at 8 for MS-SQL (works on SQL 2000 and above). Check the FreeTDS site for the value mapping if you are unsure.

Note: Make sure your SQL server is listening on the port you assign (default is 1433). On SQL 2005 and above, you may need to turn off dynamic ports.

You can test accessing the SQL server with the ‘tsql‘ executable in the freetds/bin directory. This is basically a telnet test to the SQL server. Once you have tested this successfully, you should be able to use the PHP mssql_* commands in your scripts!

Now, one final thing. If you reboot the system, you’ll notice Apache doesn’t auto start. That’s because compiling this way didn’t add any startup scripts.  You’ll need to create one to fit your distribution as necessary.

Hope this helps someone out there! I know it’s been extremely useful for me.

(Note: This guide was originally published 2009-11-04 and has not been updated)

Here’s a basic guide to installing an inbound mail filter using Postfix, Amavisd-new, and ClamAV on Ubuntu 8.04 or 8.10, that forwards to an internal mail system (Exchange for example).

Install Ubuntu on your machine (I highly recommend doing this on a VM or spare box). You can get Ubuntu here. Note that 9.10 was just released, so you’ll want to get 8.04LTS.

Follow the install defaults, and when you get to package installation, just install OpenSSH for now; we’ll deal with the other packages we want later.

Once you’ve got Ubuntu installed, log in at the console again, not via SSH yet. First off, update the package list: sudo apt-get update

Now install the build-essential package: sudo apt-get install build-essential

Once that’s finished let’s apply all the system security updates: sudo apt-get upgrade. You should notice two packages held back with names like this: linux-kernel-2.x.x.x. This is normal; kernel upgrades have to be explicitly specified.

Okay, now you’ve got an up to date system. Lets install the programs we are going to use.

Install Postfix: sudo apt-get install postfix

Postfix will ask you a couple questions; install as an Internet Site, and use the local system name.

Install SpamAssassin: sudo apt-get install spamassassin

Install Pyzor and Razor: sudo apt-get install pyzor razor

Install Amavisd-new: sudo apt-get install amavisd-new

Now, we need to install ClamAV, but the default repository for Ubuntu 8 doesn’t have the latest release; fortunately some generous souls have backported it from Ubuntu 9. So lets enable the backports repository.

Run sudo vim /etc/apt/sources.list and find the two lines that contain intrepid-backports. There’s usually a block of text above each set of sources that explains what it is. Save and quit the vim editor.

Now, you’ll need to refresh apt’s cache again, so run sudo apt-get update to update the package lists. This will add in the locators for the backported packages.

Install ClamAV: sudo apt-get install clamav clamav-daemon

We also need some decompression libraries so ClamAV can scan inside compressed files.

Install libraries: sudo apt-get install arj bzip2 cabextract cpio file gzip lha nomarch pax rar unrar unzip zip zoo

(By now you’ve probably figured out that you could install all this stuff with one command if you listed all the package names…)

Now, the default configuration of ClamAV and Amavisd from the Ubuntu repositories are pre-configured to work with each other (thanks to whoever put those packages together!). We do need to make a few changes…

ClamAV and Amavisd need to be able to access each other’s files:

Run: sudo adduser clamav amavis

Run: sudo adduser amavis clamav

(Under Ubuntu, each user also has a group with the same name.)

Okay, now we have our programs installed, lets get them doing something!

First off, lets configure Spamassassin:

Run: sudo vim /etc/default/spamassassin and change the ENABLED=0 to ENABLED=1. This enables daemon mode, which is much more memory efficient.

Run: sudo vim /etc/spamassassin/local.cf and uncomment lock_method flock. We are going to keep a local database. NOTE: If you are using an NFS mount to store the data do NOT uncomment this line. You’ll fry your database. If you’re using NFS and reading this guide though, you probably should already know this.

Since we are using Amavis to do the program linking, the majority of the configuration of Spamassassin is there, rather than in Spamassassin’s configuration files. So, lets dig into Amavisd.

Now, the set up of Amavisd under Ubuntu uses split configuration files, as opposed to previous versions and other distributions which use a single monolithic file. Either way works, but the split file method does allow for easier overridding of defaults. If you look in /etc/amavis/conf.d/ you’ll find the configuration files. The higher numbered files override the lower number files, so generally your customizations should be placed in 50-user.

That said, I find it easier to do some customizations directly in the default files. Just remember to document them so you can upgrade to newer versions easily without trying to diff everything!

First off, sudo vim 05-domain_id. Change the local_domains_acl line to contain yourdomain.ext, where that is the domain you are going to receive mail for. If you have multiple domains to receive mail for, separate each domain with a comma, and surround each domain name with single quotes. Save and quit the file once this is done.

Now lets edit the 20-debian_defaults file: sudo vim 20-debian_defaults. Most of the items in here we are going to override later, but there is one section I generally edit here: banned files. Scroll down through the file until you see a section with a banned files comment, and look for a line similar to this: qr’.\.(exe|vbs|pif|scr|bat|cmd|com….)$’i, #banned extension – basic. Add any additional file types you want to ban from being received via e-mail. Separate each extension with a pipe. Save and quit the file once you are happy with it.

Now for the meaty bit, 50-user. Open it for editing: sudo vim 50-user. First thing to do is set our spam kill levels:

$sa_spam_subject_tag = ‘***SPAM*** ‘;
$sa_tag_level_deflt  = undef;  # always add spam headers
$sa_tag2_level_deflt = 3.5; # add spam to subject tag and forward message
$sa_kill_level_deflt = 5; # quarantine message and do not forward
$sa_dsn_cutoff_level = 10;   # spam level beyond which a DSN is not sent

This always adds the spam headers to each message so you can see the analysis done by Spamassassin, even if the message is not flagged as spam. This is useful for tracking down false positives and negatives.

Now, we don’t want to bounce e-mails with banned extensions, sometimes we may actually want them. Add this line:

$final_banned_destiny  = D_DISCARD;

This ‘discards’ the message without sending an NDR to the sender. D_DISCARD also means the message is quarantined on the server, not actually deleted.

Depending on how busy the mail server will be, you may need more than the default number of amavisd-new processes. I usually configure 4 processes and find this is enough for most servers.

$max_servers = 4;

There are also notification options for when an email containing a banned file or virus is blocked. You can set these or not as you wish:

    $virus_admin = “administratorname\@example.com”; #administrator address for virus block notifications
$warnvirusrecip = 1;  # notify recipient of virus
$warnbannedrecip = 1; # notify recipient of banned file type

$mailfrom_notify_admin = “filter\@mailserver.example.com”; #identifying email address for warning messages to administrator
$hdrfrom_notify_sender = “filter\@mailserver.example.com“; #if you send sender notifications, we usually don’t
$hdrfrom_notify_admin = “filter\@
mailserver.example.com“;

Now, Amavisd-new is configured, lets get Postfix configured. Postfix configuration is done in two key files: main.cf and master.cf.

First off, lets work with main.cf: sudo /etc/postfix/main.cf

I like to change the email banner announcing the server to identify less information: smtpd_banner = $myhostname ESMTP

I also disable TLS, unless you are going to actually set up certificates: smtpd_use_tls = no

Set the hostname: myhostname = MAIL1.EXAMPLE.COM

Set the origin domain for mail from this server: myorigin = EXAMPLE.COM

Since we are going to forward to an internal mail system set: mydestination =

Yes, it should be blank.

Set the domains you are going to accept mail for and send to the internal system: relay_domains = example.com example.org

You need to tell Postfix how to get to your internal mail server. You can either do this with MX records, if it can access your internal DNS system, or you can hard-code the relay address. Hard-coding works for most setups with a single server: transport_maps = hash:/etc/postfix/transport. We’ll create the transport file later.

If you are putting this server behind a NAT device, Postfix will need to know what external IP address it is sending as: proxy_interfaces = EXTERNAL.IP.ADDRESS.HERE

Now, Postfix needs to know how to send messages to Amavisd for filtering: content_filter = smtp-amavis:[127.0.0.1]:10024

Lets not tell people how we are selecting valid users: show_user_unkown_table_name = no

Now, you need a way to reject email addresses that aren’t valid. I use a script that generates a list of valid addresses from my Exchange server. You’ll need something similar to this: relay_recipient_maps = hash:/etc/postfix/valid_recipients. The valid_recipients file should contain e-mail addresses with the format: user@example.com OK to accept mail destined for that address.

Now, lets set up some filtering to block spam before it even gets to SpamAssassin:

   smtpd_client_restrictions =
permit_mynetworks,
permit_sasl_authenticated,
reject_rbl_client zen.spamhaus.org=127.0.0.2,
reject_rbl_client zen.spamhaus.org=127.0.0.4,
reject_rbl_client zen.spamhaus.org=127.0.0.5,
reject_rbl_client zen.spamhaus.org=127.0.0.6,
reject_rbl_client zen.spamhaus.org=127.0.0.7,
reject_rbl_client zen.spamhaus.org=127.0.0.8,
reject_rbl_client zen.spamhaus.org=127.0.0.10,
reject_rbl_client zen.spamhaus.org=127.0.0.11,
permit

This uses the SpamHaus ZEN RBL to block systems that shouldn’t be sending mail, or are known spam/malware/viral senders. MAKE SURE YOU READ THE SPAMHAUS TERMS OF USE. Larger mail servers will need to pay for this service.

And a couple more filters to weed out bad servers:

smtpd_recipient_restrictions =
permit_mynetworks
reject_unauth_destination
reject_unauth_pipelining

smtpd_sender_restrictions =
permit_mynetworks
reject_non_fqdn_sender

Now, I translate mail for local UNIX services to send to my administrator account:

   local_transport = virtual
local_recipient_maps = $virtual_mailbox_maps

   virtual_alias_maps = hash:/etc/postfix/virtual

Now, save the file and quit the editor.

Okay, almost done! Lets configure the Postfix services. Remeber that content_filter line we added? We need to define that now.

Open up master.cf: sudo vim /etc/postfix/master.cf

Look for the line that contains pickup. Add these lines right under it:

-o content_filter=
-o receive_override_options=no_header_body_checks

Now scroll down to the end of the file and add:

smtp-amavis     unix    –       –       –       –       4       smtp
-o smtp_data_done_timeout=1200
-o smtp_send_xforward_command=yes
-o disable_dns_lookups=yes
-o max_use=20

127.0.0.1:10025 inet    n       –       –       –       –       smtpd
-o content_filter=
-o local_recipient_maps=
-o relay_recipient_maps=
-o smtpd_restriction_classes=
-o smtpd_delay_reject=no
-o smtpd_client_restrictions=permit_mynetworks,reject
-o smtpd_helo_restrictions=
-o smtpd_sender_restrictions=
-o smtpd_recipient_restrictions=permit_mynetworks,reject
-o smtpd_data_restrictions=reject_unauth_pipelining
-o smtpd_end_of_data_restrictions=
-o mynetworks=127.0.0.0/8
-o smtpd_error_sleep_time=0
-o smtpd_soft_error_limit=1001
-o smtpd_hard_error_limit=1000
-o smtpd_client_connection_count_limit=0
-o smtpd_client_connection_rate_limit=0
-o receive_override_options=no_header_body_checks,no_unknown_recipient_checks

Remember my comment about the number of amavis-d processes? The max_servers number from Amavisd’s configuration files needs to match the maxproc column in Postfix’s master.cf. This is the second from last column in this file (there’s a 4 above already, if you’ve copied and pasted my settings from this article, you don’t need to change anything.)

Save the file and quit. Postfix configuration is done, we just need to create the data files we specified!

Lets set up the transport table first, so Postfix knows where to send stuff:

sudo vim /etc/postfix/transport

Create lines with the following format, one per domain: example.com   smtp:[IP.ADD.GOES.HERE]

In case you are wondering, placing an IP address in square brackets means to bypass the MX record lookup and just connect to that address.

Save the file and quit the editor once you have your entries.

Now, anything you specify a hash table in Postfix, you need to create the actual database file Postfix reads. Run sudo postmap /etc/postfix/transport. This will create the transport.db file, which Postfix will use for it’s lookups.

Do the same for the virtual file: sudo vim /etc/postfix/virtual. Add local users and the email address to redirect to:

root    administrator@example.com
amavis    administrator@example.com

Save and quit, then run the postmap command against the virtual file.

We’re pretty much done. Restart the server for expediency; you could restart each service, but since this is a new build, lets be lazy. sudo shutdown -r now.

Your server should be set up and ready to filter mail! Too bad Spamassassin doesn’t know anything about the kind of spam you receive. What can you do about this? Well, if you have a collection of spam email, you can train SpamAssassin with them. You do this with the sa-learn command. For example, if you have a number of spam messages in mbox format, you can train them with the following command:

sa-learn –spam –mbox –progress /path/to/spam.box

IMPORTANT NOTE: With this server, you MUST do the sa-learn as the amavis user. Otherwise it will not get imported correctly. Also, the messages must not be forwarded or have their headers otherwise mangled; Spamassassin relies on the original headers.