Navigation

Archive | Exchange

Adventures in Load Balancing: Kemp (@KempTech)

By on July 29, 2011 in Exchange, Windows

My current project is a migration from Exchange 2003 to Exchange 2010. We wanted to load balance our CAS servers and do some SSL offloading. I have never worked with a load balancer before – pretty cool stuff. We have been using a couple of Kemp 2600’s in an active passive configuration. The Kemp devices have a nice price point and seem to have all the functionality that we need. Plus the support has been excellent. They have really helped us get up and running. Things I have learned while implementing these devices:

  • You actually set the CAS servers IP gateways to the load balancer. I guess the device acts like a router when it receives new traffic that did not originally pass through the device.
  • Clients and servers can not be in the same subnet if you want to use Layer 7 transparency. Traffic will hit the load balancer and it will pass it along to the server. The server will see that the traffic originated on the same subnet, and it will send the return straight back to the server, not through the load balancer. Timeouts result.
  • The documentation repeatedly refers to “clients”. A “client” can be a workstation, but it can also be a service.  Our BES server was connecting to the CAS to find the “/Autodiscover/Autodiscover.xml” info. Since it was on the same subnet as the CAS servers, they replied back directly and not through the load balancer. Timeouts again.
  • I really like the idea of a drain stop. I can move all traffic to one CAS and work on the other.
  • We ended up turning off Layer 7 transparency since we have all servers on the same subnet. The only other real choice would be to move the load balanced servers to their own subnet. The loss of transparency means that all connections seem to originate on the load balancer. So logs become pretty useless. Trouble shooting will occur on the Kemp. We can always ssh in and run a TCPDUMP.
Now I need to find other cool things we can do with these cool Kemp boxes.

Mailbox moves and error no. c1034ad6

By on March 22, 2011 in Exchange

We have been moving mailboxes to a new Equallogic iSCSI SAN volume. The old information store is almost empty, except for one mailbox that did not get removed at the end of the move. It was sitting there with a red “X”. When we tried to delete it we received the error:

The Operation cannot be performed because this mailbox was already reconnected to an existing user.
ID no. c1034ad6
Exchange System Manager

I found this article, that suggested the it should be automatically removed once the “Exchange maintenance” runs. Since our maintenance runs over the weekend, and our “Keep deleted mailboxes for (days)” is set to 1 days, two weeks later, the mailbox should have been removed. But it wasn’t.

The problem ended up being that we had the check box “Do not permanently delete mailboxes and items until the store has been backed up” checked. This is great, but since there weren’t any mailboxes left in the information store, we had stopped backing it up!!!!

If you get the error above, and

  • the lat mailbox has been moved
  • you have stopped backing up your information store
  • The check box “Do not permanently delete mailboxes and items until the store has been backed up” is checked

Then your last mailbox will never be permanently deleted!!!

Obvious, but maybe someone will run across this and it will save them a couple of minutes scratching their head!

Verify if a Recipient Policy is being applied to a user

By on March 11, 2011 in Exchange

We use the excellent Dell EMS Email Continuity product. It is fantastic. All our mail is replicated (via an SMTP sink and vaultbox) to their data center. In the event of and outage, our mail will route to their system, and they provide a way for the mail to be delivered to the user. One of the benefits, is that we only need to keep a limited about of mail on our exchange server (the rest in their archives).

We purge any mail that is over a year old. Maybe it is radical, but it works for us. Users prefer a snappier exchange server to messages from 10 years ago.

I do this using Recipient Policies. I can apply “Delete immediately” for messages older than a certain date, to users in AD groups. For example, I use the following in my filter rule:

memberOf=CN=MailBoxPurge-12M,DC=DOMAIN,DC=LOCAL

To verify that a user is receiving the policy, I do the following:

  1. open adsiedit.msc
  2. In the “Select a well known Naming Context”, select Configuration
  3. Navigate to Services -> Microsoft Exchange -> Organization Name -> Recipient Policies -> Select the properties of the Recipient Policy that you want to verify
  4. Obtain the objectGUID
  5. Switch the “Select a well known Naming Context” to Default Context
  6. Now navigate to employe and select properties
  7. You will see a value msExchPoliciesIncluded.

If the GUID you looked up in step 4 is in there, then you are good to go.

My adventures in Exchange Log Replaying.

By on March 8, 2011 in Exchange

I wanted to restore our exchange environment in an off line VM. One of my goals was to take an information store and mount it – just to prove that I can access the data.

At our off site business continuity location, we have a Double-Take replica of our production Exchange 2003 server, including replicas of all our information stores. Those replicated store are on an Dell Equallogic SAN, which has snapshot capabilities.

I figured I could take a snapshot of the volume that contains the information store I want present it to my offline exchange, and just mount the store. When I tried to mount the store, I realized this would not be be easy. Time to do some learning.
First Error:

The log version stamp of logfile Drive:\PATH\mdbdata\E01.log does not match the database engine version stamp. The logfiles may be the wrong version for the database.

This one was easy. I was too lazy to install SP2 on the recovery server. Seems that I needed to as the logs are stamped with a version number. Quick Fix – install Exchange 2003 Sp2.

Next error:

ESA: The Exchange Virtual Server needs to be upgraded before coming online. From the Cluster Administrator program, select ‘Upgrade Exchange Virtual Server’ from the resource’s context menu to upgrade this Exchange virtual server.

Again another easy one. I just had to go into the cluster admin and right click “Upgrade Exchange Virtual Server”

Next error:

An internal processing error has occurred. Try restarting the Exchange System Manager or the Microsoft Exchange Information Store service, or both

I was using a Standby Exchange Cluster as described here. This error suggested that I did not have al my drives and paths right. For some reason this inherited Exchange 2003 cluster had the exchange program files installed to an “E:” drive. I added another drive and re-installed exchange with Sp2.

Next error:

Database recovery/restore failed with unexpected error -515.

Alright! This was a good one. I suspected my snapshot of a replicated information store might not be happy. How do I make it happy? So begins my adventures in replay logs. Here is the summary of what I learned. I understood how what to do, I just did not know how to do it!

If you suspect you are having an issue with an information store that you can not mount, run (here is your only warning – backup before you try anything.) :

eseutil.exe /mh MailStore.edb

If you see: State: Clean Shutdown – there is something else going on.
If you see: State: Dirty Shutdown – you need to get the mail store to a clean state.

AFIAK, you can get a mailstore to a clean state one of two ways. The first is to replay the missing logs and the second it to repair. The second results in loss of data.

To replay logs, look at the output of the above command again and you will see :
Log Required: 30012-30013 (0x753c-0x753d).

If you have these log files (they will have a prefix like E00 or E01), all you should need to do is run 

eseutil.exe /r E01

This will find the files and replay them into the MailStore.

I tried this but I was missing a log file. I received the following error:

Operation terminated with error -515 (JET_errInvalidLogSequence, Timestamp in next log does not match expected) after 4.16 seconds.

Since I was missing a log file, and I had no other option to get it (like from a backup (this is all dev so there was need no to panic)). I had to resort to the “I loss data” method:

eseutil.exe" /p MailStore.edb

This took several hours and the output looks like this:

Initiating REPAIR mode…
Database: MailStore.edb
Streaming File: MailStore.STM
Temp. Database: TEMPREPAIR3692.EDB

Checking database integrity.

The database is not up-to-date. This operation may find that
this database is corrupt because data from the log files has
yet to be placed in the database.

To ensure the database is up-to-date please use the ‘Recovery’ operation.

Scanning Status (% complete)

0 10 20 30 40 50 60 70 80 90 100
|—-|—-|—-|—-|—-|—-|—-|—-|—-|—-|
……………………………………………

Scanning the database.

Scanning Status (% complete)

0 10 20 30 40 50 60 70 80 90 100
|—-|—-|—-|—-|—-|—-|—-|—-|—-|—-|
……………………………………………

Repairing damaged tables.

Scanning Status (% complete)

0 10 20 30 40 50 60 70 80 90 100
|—-|—-|—-|—-|—-|—-|—-|—-|—-|—-|
…………………….
Deleting unicode fixup table.
……………………..

Repair completed. Database corruption has been repaired!

Repaired!

Now when I run:

eseutil.exe /mh MailStore.edb

I see “State: Clean Shutdown” and I can mount my MailStore. That was fun.

Exchange 2010 SP1 and New-DatabaseAvailabilityGroup

By on March 4, 2011 in Exchange, PowerShell

I was progressing along on with my offline Exchange install, and I ran into a problem when creating my Database Availability Group (DAG). I wanted to put the witness directory on a non exchange server and the instructions say:

If the witness server you specify isn’t an Exchange 2010 server, you must add the Exchange Trusted Subsystem universal security group to the local Administrators group on the witness server.

I added the correct group to the correct group and I run:

New-DatabaseAvailabilityGroup DAGNAME -witnessserver nonexchange.domain.local -witnessdirectry c:\DAGFSW

I received the following error:

WARNING: The Exchange Trusted Subsystem is not a member of the local Administrators group on specified witness server nonexchange.domain.local.

But it is in there, believe me I tripple checked. I also tried:

  • I rebooted the server, deleted the DAG and tried again. Nada.
  • I added the witness machine$ account to the local Administrator’s account – since all it is doing is creating a shared directory. Nope.
  • Looked on the witness server, and did not see a shared folder.
  • But, I never added a member to the DAG because I thought that the shared folder should be there.

So I started reading and I came across this article. Devin suggests that all you need to do, like the documentation says, is add the Exchange Trusted Subsystem to the local administrators group, and NOT add the witness machine$ account to the Exchange Trusted Subsystem group. I agree with his argument as to why it is not necessary.

BUT. I think there might be a bug in SP1. My findings are:

  • If you run the New-DatabaseAvailabilityGroup command and ONLY have the Exchange Trusted Subsystem as a member of the witness’s local Administrators group:
    1. You will receive this error: WARNING: The Exchange Trusted Subsystem is not a member of the local Administrators group on specified witness server nonexchange.domain.local.
    2. If your witness folder is a directory or two deep, parent directories will be created
    3. The witness shared folder will not be created until you add a member to the DAG
  • If you run the New-DatabaseAvailabilityGroup command AND have the witness machine$ account in the Exchange Trusted Subsystem:
    1. You will NOT receive an error.
    2. The witness shared folder will not be created until you add a member to the DAG

So, in summary, it seems:

  • That there is a bug in SP1 in the New-DatabaseAvailabilityGroup command. It incorrectly reports that “The Exchange Trusted Subsystem is not a member of the local Administrators group”, when it is.
  • New-DatabaseAvailabilityGroup creates the DAG and even though it spits back an error, everything seems to function once a DAG member has been created – the witness folder is created
  • Devin’s article is still a valid recommendation as you do not need to add the non exchange witness machine$ account to the Exchange Trusted Subsystem group to get a DAG up and running.

Of course there could be other things at play, but as of now, this is what I have found.

My offline Exchange 2010 install script

By on March 3, 2011 in Exchange, SCCM

As I mentioned in my previos two posts, I am creating an offline replica of our Windows environment in order to test migrating from Exchange 2003 to Exchange 2010.  Now that I had a replica of our Exchange 2003 server running, I wanted to add a new Exchange 2010 box. I already had a SCCM OSD with 2008 R2 that installs the box and adds it to the domain (in this case it is the fully functioning offline domain). I added the new SP1 to meet all the prerequisites. Then I used the following commands:

#In powershell (I had to do it twice for some reason)
Import-Module ServerManager
Add-WindowsFeature NET-Framework,RSAT-ADDS,Web-Server,Web-Basic-Auth,Web-Windows-Auth,Web-Metabase,Web-Net-Ext,Web-Lgcy-Mgmt-Console,WAS-Process-Model,RSAT-Web-Server,Web-ISAPI-Ext,Web-Digest-Auth,Web-Dyn-Compression,NET-HTTP-Activation,RPC-Over-HTTP-Proxy -Restart

Set-service NetTcpPortSharing -startuptype automatic

.\setup.com /PrepareAD
.\setup.com /mode:install /roles:mb,ht,ca

Creating an offline replica of our Windows Environment-Part 2

By on February 28, 2011 in Exchange, VMware, Windows

In part one I setup an offline DC with DHCP, DNS, AND and a Windows 7 VM. Next I wanted to create a copy of our exchange server offline. Easy, right? Just install exchange 2003 with DisasterRecovery mode (setup.exe /DisasterRecovery). Easy.

That is when the fun started.  I brought up one VM with Server 2003 R2 Enterprise, set the server name to the same as the production server EXCHANGESERVER01 (this is all offline so we are okay) and tried installing with the DisasterRecovery switch. Error.:

The component “Microsoft Exchange Messaging and Collaboration Services” cannot be assigned the action “Disaster Recovery” because: – The server object for this server (“EXCHANGESERVER01”) is a clustered Exchange Virtual Server . You may not perform maintenance on this object from a standalone server.  Cluster Admin should be used to perform maintenance on this Exchange Virtual Server.

Crap. Our production Exchange 2003 server is a cluster, and the installer found the AD record (offline) for the mail server. At least we know know what that the correct info is in AD! (note to self, look at EXCHDUMP to find the info that AD has). So I decided to try and trick Exchange 2003 that is being installed on a cluster, but with only one node.

So I added a couple of drives to the 2003 server (and a rename) and changed the ESX SCSI controller to “Virtual”. Tried to boot the VM and . . . Error. Crap.

Power On virtual machine:VMware ESX Server cannot open the virtual disk, “/vmfs/volumes/GUID/OEXCHANGESERVER01/EXCHANGESERVER01-02.vmdk” for clustering.  Please verify that the virtual disk was created using the ‘thick’ option.

Turns out you have to create the VMDK drives images from the command line if you want them to act like a cluster and be shared!

vmkfstools -d eagerzeroedthick -c 1G -a lsilogic EXCHANGESERVER01-02.vmdk

I recreated all my drives, and got the VM to boot. Next, I went into Cluster administrator and setup a simple 1 node cluster. Now it was time to try and install exchange 2003 AGAIN. I fired up Exchange’s setup.exe /DisasterRecovery, AGAIN. Error. Crap.

The component “Microsoft Exchange” cannot be assigned the action “Disaster Recovery” because: – Microsoft Exchange setup does not support the use of the DisasterRecovery action when running on cluster nodes.

Seems like I am in a loop. Can’t install exchange via DisasterRecovery on a cluster, and I can’t install exchange on a single machine that has an AD record of it being a cluster. Crap.

I decided to just go ahead and install exchange normally, no DisasterRecovery switch, on the one node cluster. While I went through the setup, I did some surfing. Turns out what I was trying to do is :

How to Move All Exchange Virtual Servers from a Production Exchange 2003 Cluster to a Standby Exchange 2003 Cluster.

Maybe it is just me, but that title does not sound like what I am trying to do – recover a cluster on a single  node. Anyway, I continued installing exchange 2003 normally, on my offline single node cluster. Applied service pack 2, and then added a couple of roles to the cluster acording to the article.

  1. Created an IP address and Network name on the cluster that matched the production server, and brought them online
  2. Created an Exchange System Attendant resource.

Once I created an Exchange System Attendant resource, the Cluster Administrator tool found the AD records and everything was matched up.  All the Exchange system configuration was there. ESM looked like it did on our production servers. Perfect. I borough one of our Information stores online (no data so it just creates a new one)

Now in theory, I can launch Outlook on my Windows  7 VM, and it should create a new mailbox in the right store (for this test, I did not try to restore our edbs – that is my next post.) .

Worked. Very nice.

 

 

Creating an offline replica of our Windows Environment-Part 1

By on February 25, 2011 in Exchange, VMware, Windows

I am working on my new project, upgrading our Exchange 2003 environment to Exchange 2010. I wanted to create an off line replica of our current environemnet. This is how I set about doing that

  • Before I started, I created a new workstation – Windows 7 on our ESX server. This machine is joined to the domain, but no one has ever logged on (deployed via SCCM OSD). I left it overnight to make sure it was in AD and AD had replicated.
  • Next I created a network in ESX that is not attached to any adapters (and no other machines are connected to) – called “offline network”
  • I then used the built in ESX function of cloning a VM, and created a clone of one of our Domain Controllers (DC).
  • This clone was assigned to the off line network, and the Windows 7 VM was moved to the offline network.
  • Since the DC has AD,DNS, and DHCP all running on it, I should be able to reboot the Windows 7 machine and I should be able to log in with my account.
  • Since I never have logged on to this Windows 7 VM before, I know it is not using a cached version of my account and the Windows 7 VM has to be communicating to the DC.
  • Finally, I seized the FSMO roles from the the other non existant DC (not on the offline network)

That gets me a functioning DC and a functioning DNS with a test Windows 7 machine and Office 2010.

Next up: Recovering an Exchange 2003 cluster to a single machine in an off line network.

Powered by WordPress. Designed by WooThemes