Analyzing hybrid migration logs using PowerShell

While I’m currently working on migrating a customer from an on-prem Exchange environment to Exchange Online, in ran in to some problems with a few mailboxes.

In this case, there were three mailboxes that would fail the first (staging) sync from on-prem to ExO, due to the infamous ‘bad item limit reached’ error. So, I increased the bad item limit for these mailboxes and resubmitted the move request. After some time, the migration failed again, with the same error. The number of bad items had increased to above the limit I had set before. So, time to do some further digging. First, i’ll do a selection on the move requests to see which requests actually did fail.

1

$statistics = get-moverequest | where {$_.status -eq 'failed'} | Get-MoveRequestStatistics

I get the move requests that have a status of ‘failed’, get the statistics for those requests and load them to the variable $statistics.

Let’s see what the current amount of ‘bad items’ is for these mailboxes

1

$statistics | select-object Displayname, Item

An example from the output for one of the three mailboxes (please note that part of the displayname is hidden in this picture):

As you can see, I previously set the bad item limit to 700, but the migration currently encountered 788 bad items and therefore failed. I always do expect some bad items to occur during these migrations, but this sure is a lot. Where do all these errors come from? To find out, we have to take a look at the actual migration report.

Because I was looking at the third failed mailbox in my list of failed mailboxes, I’ll request the statistics for this mailbox, including the migration report.

1

Get-MoveRequestStatistics $statistics[2].Alias -IncludeReport | fl

This returns a huge wall of text, including all the errors that were encountered moving the messages. One of the last lines is the last failure recorded in the move request.

Of course, you can export this report to a text a file to go through the items to find the root cause. Personally, I find it easier to export the report to an XML-file, so I can use PowerShell to do some further digging.

1

Get-MoveRequestStatistics $statistics[2].Alias -IncludeReport -Diagnostic -DiagnosticArgument verbose | Export-Clixml 'C:\Users\365dude\temp\statistics.xml'

With this cmdlet, I take the statistics for the given user, including the report, and export it to the given file. Next, I can import this XML-file to an object in PowerShell.

1

$report = Import-Clixml 'C:\Users\365Dude\temp\statistics.xml'

I now have the $report variable, which holds the XML-file with the migration report. I can now navigate through this report as I could with any other XML object within PowerShell. The LastFailure entry I mentioned earlier, for example, is in fact an entry in the XML.

1

$report.LastFailure

So, can we extract some actual info from these bad items from the report? We can. The encountered failures are located in the actual report, in the failures section.

1

$report.Report.Failures

Again, I obfuscated the folder name in this screenshot. This is just a part of the output from the above command, all encountered errors will be listed in the output.

So, let’s see if we can find some common denominator in these errors. I’d like to see all errors, but just a few properties for each error.

1

$index=0;$report.Report.Failures | foreach { $_ | Select-Object @{name="index";expression={$i}},timestamp,failuretype,Message;$index++} | ft

Because there is no index number for the entries, I add one manually. That way, I can always look up a specific error by referencing the number. As arrays start to count at zero, I do the same for my index number. For each error in the file, I then select the given index number, the timestamp, failuretype and the error message. At the end, I increase the index number with one, so the next error will have a correct index.

For the mailbox in our example, this gives the following output:

So there you have it: it seems the mailbox has some items that probably have access rights mapped to an non-existing user. Of course, we can check this from the Exchange Management Shell. In this case, some of the errors referenced items in a subfolder of the ‘verwijderde items’ folder, which is Dutch for ‘Deleted Items’. So, i’ll get the folder permissions for this folder.

1

get-mailboxfolderpermission 'username:\verwijderde items\intranet'

And indeed it does show a lot of non-existing, previously deleted, users.

So in this case, I can resolve the issue by removing the legacy permissions and restarting the job. You can also decide, after reviewing the report, to restart the job with the badItemLimit paramater increased to a number high enough the not cause the move request to fail, because these errors indicate that although the permissions will not be migrated, the items itself will be copied to Exchange Online so no data will be lost.

In conclusion, you can see why I prefer to review the errors in an Exchange hybrid migration using the export-clixml cmdlet. It is a much more convenient way to navigate around all errors and get a complete view of the issues.