Exchange 2010 had many new enhancements and improvements over prior versions of Exchange. One really cool feature was the introduction of the Calendar Repair Assistant (CRA). The CRA is a mailbox assistant that is configurable through the Exchange Management Shell and runs within the MS Exchange Mailbox Assistants service. Its intended purpose is to help maintain consistency of calendar meetings between an organizer and the attendees by comparing the meeting copies of the organizer and the attendees. Why did the Exchange Product Group build this really awesome component into Exchange 2010? I’m glad you asked. I want to take you on a journey back to…The Good Ole Days!
The Good Ole Days
In the “Good Ole Days” (or even as recently as last week), the Exchange support team logged numerous calls and cases on calendar meeting issues for prior versions of Exchange. Because of the flexibility allowed in terms of what end users can do with meetings in their calendars, meetings can become inconsistent across organizers and attendees calendars. These issues would range from inconsistent meeting times between organizers and attendees to meetings being “unknowingly” deleted from a user’s calendar. We saw cases where meetings would show up in Outlook but not a mobile device or vice versa. Sometimes meetings would be duplicated on a calendar or would lose their organizer. Delegates would reportedly make a change to a meeting but it wouldn’t show up on the mailbox owner’s mobile device. The list goes on. The troubleshooting of these issues, though normally pretty straight forward can be tedious and time consuming. I went ahead and included a troubleshooting reference guide for you below not only to point out how we would troubleshoot and identify these problems, but also just in case you’ve stumbled onto this blog and you’re experiencing something similar!
Exchange calendar troubleshooting reference
Getting to the most recent service pack and update of your Outlook client and Exchange Server is very important as doing that might actually address your reported problems. Depending on the type of issue you are experiencing, there are several different ways to troubleshoot. If you are trying to determine why a meeting keeps changing or is moved to the deleted items folder, etc. MFCMapi is a good tool to use. You can launch MFCMapi and connect to the mailbox profile of the user that is reporting the issue. Find the calendar item that you are looking for and save out the properties of that item to a text file. Search for the PR_LAST_MODIFIER_NAME, PR_LAST MODIFIED_DATE, PR_SENDER_EMAIL, and PR_SENT_REPRESENTING_NAME. These properties will paint a pretty good picture for you. Often times we’ve seen that the last user to modify the item is either a service account for a MAPI/CDO device, a delegate, or the mailbox owner themselves.
Another great tool you can use to see what is happening to a meeting is Exchange Trace or “Trace Control” which can be found in the Exchange Troubleshooting Assistant in Exchange 2007 and 2010. You can setup your trace, check all of the boxes under “Trace Types” and then on the “Components to Trace” you will select “Store” with the following “Trace Tags”: tagCalendarChange, tagCalendarDelete, tagMtgMessageChange, tagMtgMessageDelete. You can then setup a mailbox filter for the mailbox in question and kick off the trace. At this point you will need to reproduce the issue (or wait until it reproduces). Once it does stop the trace. You’ll need to get with Microsoft support at this point to convert and analyze the trace for you.
One of my colleagues over on the Outlook team, Randy Topken, created a great tool called CalCheck. It performs a wide variety of tests and is very helpful in troubleshooting Calendar issues. I won’t go through all of them as he has published everything you need to know here: CalCheck - The Outlook Calendar Checking Tool.
There are many other issues that we have encountered, but this is a glimpse of the most common types of issues we have seen (and actually continue to see) and how we go about troubleshooting those problems. Hopefully this gives you a little insight as to why we were so excited to see the Calendar Repair Assistant in Exchange 2010. I’ve included some additional links to assist you in troubleshooting calendar below:
- Working with support to troubleshoot the Outlook calendar in an Exchange environment
- How to troubleshoot calendar issues in Microsoft Outlook, Microsoft Exchange Server 2007, and Microsoft Exchange Server 2010
- How to troubleshoot missing and duplicate appointments in Outlook
- Troubleshooting Free/Busy Information for Outlook 2007
CRA RTM
Now let’s get to the heart of this article, THE CRA! The CRA was built to detect and correct calendar inconsistencies like I described above between a meeting organizer and attendees. The CRA must be configured through the Exchange Management Shell in order to run, as it is not enabled by default. Once enabled, the CRA will run against all mailboxes on the server you have configured it for. You can disable it for specific mailboxes by using the Set-Mailbox cmdlet and setting the CalendarRepairDisabled to True. In the RTM release of Exchange 2010, the CRA was a time-based assistant that was configured to run on a specified schedule and could not be throttled to adjust for server resource utilization. CRA made decisions only by comparing an organizer’s copy of a meeting with the attendee’s copy of the meeting. CRA uses the organizer’s meeting copy as the master and assumes it is the correct copy. It then checks attendee response status and assumes that the attendee’s response status is correct and updates the tracking information for the organizer. The problem was that CRA had no idea of how the inconsistencies occurred, meaning it didn’t take into account client intent i.e an attendee intentionally changes a meeting start time or location on their own calendar. One of the biggest issues we saw with this lack of client intent checking in RTM was if an attendee deleted a meeting from their calendar but did not send an update to the organizer, the CRA would recreate the meeting in the attendees calendar and would continue to do so until the attendee sent a response to the organizer (this has been corrected in SP1). When the CRA detects an inconsistency on a calendar, it issues a special meeting message called a Repair Update Message. The message is processed by the Calendar Attendant and then the message is placed in the user’s Deleted Items folder. Any repairs made are recorded in the CRA log (see Troubleshooting CRA below).
CRA SP1 and later
Exchange 2010 SP1 saw some pretty cool changes to the CRA. As mentioned earlier, in the RTM version CRA was time based and ran on a specified schedule. In SP1 and later this changed to a throttle based assistant to help prevent it from impacting server resources or end user experience. You can enable CRA with the Set-MailboxServer cmdlet. You will need to set the CalendarRepairWorkCycle and the CalendarRepairWorkCycleCheckpoint. The work cycle is the time span allocated for CRA to scan and process all mailboxes on the server i.e. if this is set to seven days, that means every mailbox on the server will be processed once every seven days. The throttling assistant calculates the slowest pace at which mailboxes can be processed by dividing the total number of mailboxes by the work cycle. There is also a built in mechanism to monitor certain resources like store and throttle the processing back if the load starts to rise. The checkpoint is the period of time in which the list of mailboxes in the CRA’s queue is refreshed during the existing work cycle, adding new mailboxes, moved mailboxes and mailboxes that have not processed yet into its queue. For example, the following command will set the CRA to check all mailboxes on the server once every seven days and refresh its work queue every day with the list of calendars that still need to process repairs within that seven day period:
Set-MailboxServer –Identity MBXSRV1 –CalendarRepairWorkCycle 7.00:00:00 –CalendarRepairWorkCycleCheckPoint 1.00:00:00
You can verify the settings for calendar repair options that are set for all mailbox servers by running the following in Exchange Management Shell:
Get-MailboxServer | fl name,*calendar*
Another great change in SP1 was the introduction of “client intent”. The CRA can now distinguish intentional versus unintentional inconsistencies between a meeting organizer and attendees. When the CRA is running against an attendee calendar, it will validate a meeting item against the organizer’s copy. If it is running against the organizer, it will validate the item against all attendees. The CRA will correct inconsistencies, but only for mailboxes on the server for which it is running. It reads from other Exchange 2010 mailbox servers, but each server enabled for calendar repair will have its own instance of the CRA running that is responsible for the mailboxes on that server. When the CRA finds an inconsistency, it goes about determining client intent by using snapshots of calendar items that are stored in something called the Calendar Version Store (CVS) and calendar metadata. Before a calendar item is changed, the Exchange 2010 copy on write functionality takes a snapshot of the calendar item and stores it along with additional metadata (like the source or the application that initiated the change) in the root of the Recoverable Items folder for a default 120 days before being purged. The Calendar Version Store maintains a historical record of these calendar item snapshots that are in the Recoverable Items folder. Client intent metadata is added to a calendar item as a named property: Name id = 0x0015 (dispidClientIntent), Named Prop GUID = 11000E07-B51B-40D6-AF21-CAA85EDAB1D0. Any non-zero Hex value represents an intentional change. Before a specific change is made to a calendar item, the snapshot is taken which preserves the existing metadata as well. When the CRA is determining client intent, it queries the Calendar Version Store. During the initial query, a special dynamic search folder is created called CalendarVersionStore in the root of the non-IPM subtree with a search scope of the IPM subtree and the Recoverable Items folder and criteria of all calendar items and meeting messages. Once this search folder is populated with the requested stored versions of a particular item from the Calendar Version Store, the CRA will go about determining client intent. It must first determine the target version of the particular item. It does this in one of two ways. The first method the CRA uses to determine the target version for client intent is when the resulting change, and not the state prior to the change, is important, ie an item is deleted. The Calendar Version store is sorted with the most recent item on top and then queried backwards. The oldest version found that is in the target state is used as the target version and it is the one whose client intent metadata is validated. Let’s say that item A is deleted and there are 4 snapshots of item A in the CVS. We will call them A4-A1. First A4 is queried and we see that it matches the target deleted state. Then A3 is queried and (for hypothetical purposes) we see it is also in the target deleted state. Then A2 is queried and the CRA determines that this snapshot is not in the target deleted state. Now the CRA will go back to A3 and review the metadata on it for client intent. Any non-zero entry on this item indicates an intentional change. The second method the CRA uses to determine the target version for client intent is when how an item transitioned into a certain state is important, ie the time or location on an attendee copy is different from the organizer’s copy. The main difference of this method is that it looks at the chain of snapshots from initial state to target state. If any snapshot in the chain does not show client intent, then the whole change is considered unintentional. Let’s say that the CRA finds that a meeting attendee for meeting B has a different time listed than the organizer. There are 5 snapshots of B in the attendees CVS, B5-B1. The CRA searches backwards (B5-B4-B3-B2-B1) and finds that B2 is the last snapshot with the same time as the organizer. It will then look at the metadata on B3-B5. If any of these have a zero entry, the entire chain is considered unintentional and will be corrected. All of the items in this chain must have a non-zero metadata client intent property value set for CRA to read this as an intentional change. In 2010 SP1, when the CRA detects an unintentional inconsistency it uses the same process (Repair Update Message) as it did in 2010 RTM. When the CRA detects an intentional location or time inconsistency, no repair action is taken on the calendar item. If there is an intentional missing organizer or attendee item, the CRA uses a special inquiry message to repair the item on the opposite calendar. For instance, if a meeting organizer deletes a meeting but doesn’t send out a cancellation, the meeting will still exist on the attendee’s calendar. The CRA will send a special inquiry message that is processed by the calendar attendant and trigger a normal meeting cancellation on the attendee’s calendar. For more information on CRA conflict detection and resolution logic see the “Conflict Detection and Resolution” section here: Understanding Calendar Repair
Caveats
- CRA does not run against resource mailboxes.
- If the meeting organizer does not enable the “Request a response to this invitation” in OWA or “Request Responses” option in Outlook, calendar repair will not take place on that meeting.
- If you set calendar repair to disabled for a specific mailbox, calendar repair will not take place for that mailbox. You can verify this with get-mailbox | ft name,CalendarRepairDisabled . If it’s set to true, then calendar repair is disabled.
- If you disable the Calendar Version Store for a specific mailbox, then client intent cannot be determined. You can verify this get-mailbox | ft name,CalendarVersionStoreDisabled . If it’s set to true, then the Calendar Version Store is disabled.
- The CRA only check calendar items for mailboxes that have the calendar attendant enabled (it is by default). This allows the Repair Update Message to be processed. See Set-CalendarProcessing.
- The primary clients that write client intent metadata are OWA 2010 SP1 and Outlook 2010. Client intent for clients utilizing EWS or EAS may be limited. See Client Application Experience.
- CRA will not change the meeting organizer, if it has been improperly changed.
Troubleshooting CRA
Troubleshooting calendar issues in 2010 can be approached in the same way as listed in the earlier troubleshooting section. There are some additional things you can look at when trying to see if the CRA processed a repair to a meeting and why. The first is the CRA log. It is stored on the mailbox server that it is running on and is located in <ExchangeInstallPath>\Logging\Calendar Repair Assistant and is in the format of CRA<date>yyyymmdd-<sequence>.<user>.log. Locate the CRA logs for specific users and then review them for any repairs that the CRA made. get-CalendarDiagnosticLog can be used to search the Calendar Version Store for a particular user and a particular item. Then you can use MFCMapi as outlined above to search the properties of the item and determine if there was positive client intent.
Conclusion
Hopefully I’ve shed a little light on the importance of the Calendar Repair Assistant. It is a critical component and has, in my opinion, reduced the number of cases in Exchange 2010 than what we saw for calendar issues in prior versions of Exchange. Troubleshooting, while you can still use the same techniques used in legacy versions of Exchange, has been simplified with the CalCheck, Calendar Repair Log, and using the Get-CalendarDiagnosticLog to pull calendar items from the Calendar Version Store for analysis on client intent cases.
Additional resources
- Understanding Calendar Repair
- Configure Calendar Repair Assistant Settings
- Managing Calendars
- Set-MailboxServer
- Set-Mailbox
- Changes to Recoverable Items Versioning in Exchange 2010 SP2 RU3
I wanted to thank Mike Manjunath, Rob Whaley and Jonathan Runyon for their review of this blog post.