So, tonight I started the last phase of one of my longest-running projects since joining Microsoft–an Exchange Online migration for a school district that I began nearly a year and a half ago. 40,000 mailboxes down and 13,000 public folders remaining.
One of the things that we recommend for Hybrid Public Folders is that you run the SyncMailPublicFolders.ps1 script daily. This can be a pain (as someone has to remember do it), and because it requires an Office 365 credential, you have two options: modify and hard code it in the script or store it in a more secure location, such as a service account’s local registry.
// I will digress here momentarily
While we don’t advocate storing passwords in plain-text in a script, I will show you how it’s done for this script for purposes of education. I also have code here that will show you how to do the same procedure, only encoding the data in the registry under the service account’s context (which is what we did in this case).
To make this change for the purpose of testing automation:
- Comment out the Credential parameter (lines 47 and 48)
- Set the CsvSummaryFile output (line 52) to a file that will just get overwritten every day.
- Add a code block for the credential (insert the following after the param() block and before the WriteInfoMessage() function)
# Password/credential $userAdmin = "tenantadmin@tenant.onmicrosoft.com" $userAdminPass = "Password123" $securePassword = ConvertTo-SecureString $userAdminPass -AsPlainText -Force $global:Credential = New-Object -TypeName System.Management.Automation.PSCredential -argumentlist $userAdmin,$securePassword
It will look something like this:
This will now allow the SyncMailPublicFolders.ps1 script to run as a scheduled task. Please, for the love all that is holy, don’t do this in production–use a more secure method to store credentials in an encrypted fashion.
Now, back to our regularly scheduled post.
//
Notes
- Every time I tell you to run something from your Exchange On-Premises organization, I mean “run this from the Exchange Management Shell in on your Public Folder server, unless otherwise specified.”
- Every time I tell you to do something in your Exchange Online or Office 365 tenant, I mean “run this from a PowerShell session connected to Office 365 FROM YOUR EXCHANGE ON-PREMISES SERVER.” You can do it otherwise, but this way will save you a lot of copying and pasting and frustration.
- If your on-premises organization is 2007, you’ll need to just connect to the Exchange Online endpoint (you won’t need to nor will you be able to install the Windows Azure AD PowerShell for the MSOL cmdlets, so use this three-liner to get connected to Exchange Online directly:
$o365Cred = Get-Credential $o365Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid -Authentication Basic -AllowRedirection -Credential $o365Cred Import-PSSession $o365Session
- If your on-premises organization is 2007, you’ll need to just connect to the Exchange Online endpoint (you won’t need to nor will you be able to install the Windows Azure AD PowerShell for the MSOL cmdlets, so use this three-liner to get connected to Exchange Online directly:
Preparing the Environment
- When I set up my customer for Hybrid Public Folders, I use a script that I put together to identify public folders with invalid characters. This has helped many customers avoid nasty scenarios. I stole the idea for the name (IDFix) from our tool of a similar name. You should run it in your environment to see what you get back. You can get it here: IDFix for Public Folders.
- Before I even began the night’s activities, I needed to make sure the SyncMailPublicFolders.ps1 script I had automated was stopped. No sense ruining things before I even start.
- The first thing I had to do is make sure that my customer had no existing Public Folder migrations. Just because I didn’t start one doesn’t mean they didn’t.
Run from both Exchange Online and your Exchange On-Premises Servers Get-OrganizationConfig | FL Public*Migration* PublicFoldersLockedForMigration : False PublicFolderMigrationComplete : False PublicFolderMailboxesMigrationComplete : False
- If you have True reported for any of those, STOP. Do not pass GO, do not collect $200. You’ll have to do some cleanup first.
Run this from your Exchange On-Premises Org.Set-OrganizationConfig -PublicFoldersLockedforMigration:$false -PublicFolderMigrationComplete:$false
- Run this from your Office 365 tenant to make sure no one has tried monkeying around with public folder migrations behind your back.
Get-PublicFolderMigrationRequest | Remove-PublicFolderMigrationRequest -Confirm:$False $PFMigrationBatch = Get-MigrationBatch | ? { $_.MigrationType.ToString() -eq "Public Folder" } $PFMigrationBatch | Remove-MigrationBatch -Confirm:$False Get-Mailbox -PublicFolder Get-PublicFolder Get-MailPublicFolder | where {$_.EntryId -ne $null}| Disable-MailPublicFolder -Confirm:$false Get-PublicFolder -GetChildren | Remove-PublicFolder -Recurse -Confirm:$false $hierarchyMailboxGuid = $(Get-OrganizationConfig).RootPublicFolderMailbox.HierarchyMailboxGuid Get-Mailbox -PublicFolder:$true | Where-Object {$_.ExchangeGuid -ne $hierarchyMailboxGuid} | Remove-Mailbox -PublicFolder -Confirm:$false Get-Mailbox -PublicFolder:$true | Where-Object {$_.ExchangeGuid -eq $hierarchyMailboxGuid} | Remove-Mailbox -PublicFolder -Confirm:$false - Now that we’re moderately confident that we can move forward, we need to make sure that we don’t exceed the folder quota for any posts that might be migrating in.
Get-OrganizationConfig | fl *quot* DefaultPublicFolderIssueWarningQuota : 1.7 GB (1,825,361,920 bytes) DefaultPublicFolderProhibitPostQuota : 2 GB (2,147,483,648 bytes)
No, that definitely won’t do.
- So, we update the Public Folder quotas.
Set-OrganizationConfig -DefaultPublicFolderIssueWarningQuota 9.5GB -DefaultPublicFolderProhibitPostQuota 10GB
- Next, it’s time to gather some data before we begin. Since my customer’s source environment was Exchange 2007, I needed to log into one of those servers and launch the Exchange Management Shell there. I like to do belt and suspenders, so I ran some extra data gathering commands. You can run this if you like. Your mileage may vary, but I’ve found it helpful in the past to have backups of your backups.
Get-PublicFolder -Recurse -ResultSize Unlimited | Export-Clixml .LegacyPFStructure.xml Get-PublicFolder -Recurse -ResultSize Unlimited | Get-PublicFolderStatistics | Export-Clixml .LegacyPFStatisticsRecurse.xml Get-PublicFolder -Recurse -ResultSize Unlimited | Get-PublicFolderClientPermission | Select-Object Identity,User -ExpandProperty AccessRights | Export-Clixml .LegacyPFPerms.xml
- But, I also grabbed our Public Folder Migration Scripts and saved them to my customer’s Exchange 2007 server in C:PFScripts (just like the document says, because I’m lazy).
- The next step is an internal relay domain with a well-known name that we can use to route mail during the transition. Run this from your Exchange On-Premises Org.
New-AcceptedDomain -Name "PublicFolderDestination_78c0b207_5ad2_4fee_8cb9_f373175b3f99" -DomainName tenant.onmicrosoft.com -DomainType InternalRelay
- Just in case we missed anything in the Public Folder names, we’re going to check and make sure. If you’re still unlucky enough to be running Exchange 2007 like this customer is, this is the command that you’ll run. Run this from your Exchange On-Premises Org.
Get-PublicFolderDatabase | ForEach {Get-PublicFolderStatistics -Server $_.Server | Where {$_.Name -like "**"}}If you’re in the modern era and have upgraded to Exchange 2010, you can use this command:
Get-PublicFolderStatistics -ResultSize Unlimited | Where {($_.Name -like "**") -or ($_.Name -like "*/*") } | Format-List Name, Identity - One of the meh things about Public Folder migrations is that Send-As permissions aren’t migrated. So, we’re going to grab them as well so we can apply them later. Run this from your Exchange On-Premises Org.
Get-MailPublicFolder -ResultSize Unlimited | Get-ADPermission | ? {($_.ExtendedRights -Like "Send-As") -and ($_.IsInherited -eq $False) -and -not ($_.User -like "*S-1-5-21-*")} | Select Identity,User | Export-Csv Send_As.csv -NoTypeInformation - Now, for the funsies. I’m going to make a new sub-directory to store my working files in. Again, for reference, I’ve saved all of the Microsoft-provided Public Folder scripts in C:PFScripts.
Run this on your Exchange On-Premises Org.cd PFScripts Mkdir C:PFScriptsPFMigrate .Export-PublicFolderStatistics.ps1 C:PfScriptsPFMigratePFStatistics.csv publicfolderserver.domain.com
- While processing, I saw this interesting bit of information:
[4/8/2017 3:31:30 AM] Enumerating folders under IPM_SUBTREE... [4/8/2017 3:42:04 AM] Enumerating folders under IPM_SUBTREE completed...13039 folders found. [4/8/2017 3:42:04 AM] Enumerating folders under NON_IPM_SUBTREE... [4/8/2017 3:42:06 AM] Enumerating folders under NON_IPM_SUBTREE completed...118 folders found. [4/8/2017 3:42:06 AM] Retrieving statistics from server PUBFOLD01 [4/8/2017 3:42:12 AM] Retrieving statistics from server PUBFOLD01 complete...13064 folders found. [4/8/2017 3:42:13 AM] Total unique folders found : 13064. [4/8/2017 3:42:13 AM] Retrieving statistics from server PUBFOLD02 [4/8/2017 3:42:20 AM] Retrieving statistics from server PUBFOLD02 complete...13050 folders found. [4/8/2017 3:42:21 AM] Total unique folders found : 13072. [4/8/2017 3:42:21 AM] Exporting statistics for 13072 folders [4/8/2017 3:42:23 AM] Folders processed : 10000. [4/8/2017 3:42:24 AM] Exporting folders to a CSV file
What am I supposed to do with this information? The script clearly says that it found 13,072 unique folders, but this screen output makes it seem like it’s only processing 10,000. And not like it ran into a problem 10,000. This seems like a value that someone thought would be enough. That doesn’t really sound like it’s going to work for my customer.
“Hey, I saw you had 13,072 public folders, but I only decided to grab 10,000 of them. Good luck!
Just let me know if you want any other advice for helping you free up your future to plan for another career.
So, I did what any consultant would do, and decided to tinker. I opened Export-PublicFolderStatistics.ps1 and searched for 10000. Found it on line 154 inside of the CreateFolderObjects() function. Made a copy of the line, commented it out, and updated the number of 100000 and saved.
![export-publicfolderstatistics]()
- Reran the command and received what I deemed to be better output.
[4/8/2017 3:47:27 AM] Enumerating folders under IPM_SUBTREE... [4/8/2017 3:57:53 AM] Enumerating folders under IPM_SUBTREE completed...13039 folders found. [4/8/2017 3:57:53 AM] Enumerating folders under NON_IPM_SUBTREE... [4/8/2017 3:57:55 AM] Enumerating folders under NON_IPM_SUBTREE completed...118 folders found. [4/8/2017 3:57:55 AM] Retrieving statistics from server PUBFOLD01 [4/8/2017 3:53:02 AM] Retrieving statistics from server PUBFOLD01 complete...13064 folders found. [4/8/2017 3:53:02 AM] Total unique folders found : 13064. [4/8/2017 3:53:02 AM] Retrieving statistics from server PUBFOLD02 [4/8/2017 3:53:09 AM] Retrieving statistics from server PUBFOLD02 complete...13050 folders found. [4/8/2017 3:53:10 AM] Total unique folders found : 13072. [4/8/2017 3:53:10 AM] Exporting statistics for 13072 folders [4/8/2017 3:53:13 AM] Exporting folders to a CSV file
- Now it’s time to generate the Public Folder map file, which iterates through the CSV you just created and figures out how to distribute the public folders into Public Folder Mailboxes in Office 365. In this case, I told the script to break the public folders at 10GB boundaries:
.PublicFolderToMailboxMapGenerator.ps1 10000000000 C:pfscriptsPFMigratePFStatisticsNew.csv C:pfscriptsPFMigratePFMap.csv
Data is returned that looks acceptable:
[4/8/2017 4:02:22 AM] Reading public folder list... [4/8/2017 4:02:22 AM] Loading folder hierarchy... [4/8/2017 4:02:24 AM] Allocating folders to mailboxes... [4/8/2017 4:02:24 AM] Trying to accomodate folders with their parent... [4/8/2017 4:02:24 AM] Exporting folder mapping...
In case you’re like me and always curious, you can look inside the PFMap.csv file (or whatever you call it, but hopefully you’re smart enough to figure that out without me explicitly pointing it out) to see what kind of data it generated:
![pfmap-1]()
As you can see, it’s got two columns–one named “TargetMailbox” (which will be the Public Folder Mailbox in Office 365 into which that particular branch of the public folder tree will be placed) and the “FolderPath” (which is, not surprisingly, the aforementioned particular branch). Sorry if I spoiled the surprise. You didn’t come here for surprises.
Starting the Migration
- Switching over to the PowerShell session that is connected to Office 365, I run the following:
cd PFScripts .Create-PublicFolderMailboxesForMigration.ps1 -FolderMappingCsv C:pfscriptsPFMigratePFMap.csv -EstimatedNumberOfConcurrentUsers 40000
I chose 40,000, since that’s the number of users in the organization that will most likely be connecting to and browsing public folders. The script returns some data:
Public Folder mailbox updates. Creating 14 Public Folder mailbox(es) and updating 0. Total mailboxes to serve hierarchy will be 20. Would you like to proceed? [Y] Yes [N] No [?] Help (default is "Y"):
Well, duh. Of course I want to continue. I didn’t stay up until 4am to cancel a migration.
[falls asleep]
Ok, I’m back. The recommended number of users per hierarchy-serving mailbox is 2,000. Since I entered 40,000 users, the script has determined that I need at least 20 mailboxes to serve the public folder hierarchy.
Carry on.
- After I hit Yes, the Create-PublicFolderMailboxesForMigration.ps1 script will go about its business of creating Public Folder mailboxes. Run this from your Exchange Online Tenant.
And now we wait.
![create-publicfoldermailboxesformigration-1]()
When it gets to the end, you can see that it created additional mailboxes:
![create-publicfoldermailboxesformigration-2]()
Onward and upward. - Run the SyncMailPublicFolders.ps1 script one last time to make sure everything is working. If you already have hybrid public folders enabled, you know how to do this. If you don’t:
.SyncMailPublicFolders.ps1 -Credential (Get-Credential) -CSVSummarFile:csvoutput.csv-Credential is your Office 365 global admin.
- Now, we have some additional information to gather. You can save this into variables. Run this from your Exchange on-premises org in the Exchange Management Shell.
(Get-Mailbox <admin user>).legacyExchangeDN | Out-File .MailboxLegacyExchangeDN.txt (Get-ExchangeServer <public folder server>).ExchangeLegacyDN | Out-File .ServerExchangeLegacyDN.txt $OAEndpoint = (Get-ExchangeServer).[0].ExternalHostNameHostnameString $OAEndpoint | Out-File .OAEndpoint.txt
That last one might be tricky if your org has multiple endpoints or if, in my customer’s case, you never updated external Autodiscover to the hybrid endpoint per our recommendation. What you’re really after is the Autodiscover endpoint (2013 or later) that Office 365 will use to contact your on-premises environment. In my customer’s environment, it’s hybrid.domain.edu, as is referenced in the OrgRelationship:
Get-OrganizationRelationship | fl TargetAutoDiscoverEpr TargetAutodiscoverEpr : https://hybrid.domain.edu/autodiscover/autodiscover.svc/WSSecurity
So, I need *that* value instead.
$OAEndpoint = "hybrid.domain.edu" $OAEndPoint | Out-File .OAEndPoint.txt
- Now, you need to switch back over to the PowerShell prompt that is connected to Exchange Online (hopefully you’re running it from your PF server like I already told you to, or you’re going to have some copying and pasting to do).
cd PFScripts $OAEndopint = gc .OAEndpoint.txt $MailboxLegacyExchangeDN = gc .MailboxLegacyExchangeDN.txt $ServerExchangeLegacyDN = gc .ServerExchangeLegacyDN.txt $Credential = Get-Credential <domainadmin user>
- Now, it’s time to create the Public Folder Migration Endpoint. Run this in your Exchange Online Organization.
$PFEndpoint = New-MigrationEndpoint -PublicFolder -Name PublicFolderEndPoint -RpcProxyServer $OAEndPoint -Credentials $Credential -SourceMailboxLegacyDN $MailboxLegacyExchangeDN -PublicFolderDatabaseServerLegacyDN $ServerExchangeLegacyDN -Authentication Basic
- Once the endpoint is created, I create the migration batch.
New-MigrationBatch -Name PublicFolderMigration -CSVData (Get-Content .PFMigratePFMap.csv -Encoding Byte) -SourceEndpoint $PFEndpoint.Identity -NotificationEmails <my email address>
It processes for a minute, and then gives me some feedback.
Identity Status Type TotalCount -------- ------ ---- ---------- PublicFolderMigration Stopped PublicFolder 14
- Time to get down to brass tacks. I cross my fingers and start the migration batch.
Start-MigrationBatch PublicFolderMigration
And without any fanfare at all–no ribbons, no streamer, no Def Leppard songs over giant speakers–my migration begins.
- Before leaving my workstation to grab a Diet Coke (to celebrate my apparent victory over Public Folders), I decided run a quick check (Run this from your Exchange Online Org):
Get-MigrationUser -BatchID PublicFolderMigration Identity Batch Status LastSyncTime -------- ----- ------ ------------ Mailbox1 PublicFolderMigration Syncing 4/8/2017 12:17:01 AM Mailbox2 PublicFolderMigration Syncing 4/8/2017 12:17:09 AM Mailbox3 PublicFolderMigration Syncing 4/8/2017 12:17:18 AM Mailbox4 PublicFolderMigration Syncing 4/8/2017 12:17:31 AM Mailbox5 PublicFolderMigration Syncing 4/8/2017 12:17:41 AM Mailbox6 PublicFolderMigration Syncing Mailbox7 PublicFolderMigration Validating Mailbox8 PublicFolderMigration Validating Mailbox9 PublicFolderMigration Validating Mailbox10 PublicFolderMigration Validating Mailbox11 PublicFolderMigration Validating Mailbox12 PublicFolderMigration Validating Mailbox13 PublicFolderMigration Validating Mailbox14 PublicFolderMigration Validating
Sweet. Diet Coke it is.
- Once all of the migration requests are underway, I lock the Public Folders so users can’t make changes. Run this in the Exchange On-Premises Org.
Set-OrganizationConfig -PublicFoldersLockedForMigration:$true
- At this point, it’s going to be watching some paint dry, but I like seeing pretty statistics, so I’ll show you some more. Run this from the Exchange Online Org.
Get-MigrationUser -BatchId PublicFolderMigration | Get-MigrationUserStatistics | FT -auto Identity Batch Status Items Synced Items Skipped -------- ----- ------ ------------ ------------- Mailbox1 PublicFolderMigration Syncing 0 0 Mailbox2 PublicFolderMigration Syncing 0 0 Mailbox3 PublicFolderMigration Syncing 0 0 Mailbox4 PublicFolderMigration Syncing 8279 0 Mailbox5 PublicFolderMigration Syncing 0 0 Mailbox6 PublicFolderMigration Syncing 0 0 Mailbox7 PublicFolderMigration Syncing 0 0 Mailbox8 PublicFolderMigration Syncing 0 0 Mailbox9 PublicFolderMigration Syncing 2874 0 Mailbox10 PublicFolderMigration Syncing 38535 0 Mailbox11 PublicFolderMigration Syncing 0 0 Mailbox12 PublicFolderMigration Syncing 3304 0 Mailbox13 PublicFolderMigration Syncing 0 0 Mailbox14 PublicFolderMigration Syncing 0 0
There's actually a bit more useful data that can be gathered, so I grabbed a few of the extra fields from Get-MigrationUserStatistics: Identity Status SyncedItemCount SkippedItemCount BytesTransferred PercentageComplete -------- ------ --------------- ---------------- ---------------- ------------------ Mailbox1 Syncing 32553 176 2.092 GB (2,246,307,901 bytes) 40 Mailbox2 Syncing 95156 0 5.021 GB (5,391,041,974 bytes) 94 Mailbox3 Syncing 82711 0 2.987 GB (3,207,776,110 bytes) 47 Mailbox4 Syncing 23625 0 8.373 GB (8,990,905,121 bytes) 20 Mailbox5 Syncing 1584 0 1.99 GB (2,136,923,781 bytes) 94 Mailbox6 Syncing 51243 0 2.562 GB (2,750,709,675 bytes) 70 Mailbox7 Syncing 17659 0 3.26 GB (3,500,029,634 bytes) 47 Mailbox8 Syncing 6472 0 7.234 GB (7,767,318,075 bytes) 94 Mailbox10 Syncing 281718 1 3.198 GB (3,434,352,125 bytes) 37 Mailbox11 Syncing 146567 0 4.09 GB (4,391,621,840 bytes) 50 Mailbox12 Syncing 102574 0 6.194 GB (6,650,300,317 bytes) 94 Mailbox13 Syncing 25702 0 3.951 GB (4,242,569,651 bytes) 59 Mailbox14 Syncing 29478 0 3.448 GB (3,702,518,318 bytes) 48 Mailbox9 Synced 2874 0 4.444 GB (4,771,626,709 bytes) 95
That gives a much better perspective on where a migration is.
When it gets to the end, you can see that it created additional mailboxes: