A few years ago, we migrated our email service to Microsoft’s Office365 cloud service. Overall, it’s been very reliable and eliminated the challenges we had hosting Exchange ourselves. It let us get to our emails using Outlook installed on Windows, any internet browser, and smartphones. Office365 also offered other Office product online (Access Web Apps, Excel, Word, etc.), SharePoint and OneDrive Business.
Unfortunately, on the morning of June 30th, we discovered:
- Delays sending and receiving emails
- Some emails were bouncing back from recipients who couldn’t validate our Office365 Exchange Server’s SMTP (protection.outlook.com) with our domain name. That meant the Exchange SMTP server was no longer considered a trusted sender of emails from the @fmsinc.com domain.
- Our use of the Office365 SMTP server to send emails with our Total Access Emailer product was also failing to authenticate against the server
The problems began the evening before. Needless to say, we aren’t happy about this experience which impacted us and our clients using Office365. Reports are that it affects Office365 customers across North America.
Contacting Microsoft, they confirmed problems with the health of their Office365 Exchange Server. Throughout the day, problems lessened but persisted. We hope the problems are resolved soon and that we’ll understand what went wrong once we overcome the immediate crises.
These are the reports we’ve received from Microsoft. We’ll keep you updated as we learn more:
Exchange Online Service Degraded
This is what the Office365 Admin portal shows for Service Health:
EX71628 – E-Mail and calendar access – Restoring Service
Jun 29, 2016 12:11 PM
CURRENT STATUS
Our investigation determined that an existing transport feature which is designed to expedite the delivery of email messages became degraded, which caused impact to email delivery for a subset of users. We’re bypassing the affected feature to restore service
User Impact
Users may be unable to send email messages through the Exchange Online service. Email messages may appear to be stuck in the Drafts or Outbox folders.
Scope of Impact
A few customers have reported this issue, and our analysis indicates that for most customers, it’s unlikely that many users would report impact related to this event.
- Start Time: Thursday, June 23, 2016, at 3:00 PM UTC
Preliminary Root Cause
An existing transport feature that is designed to expedite the delivery of email messages became degraded, which caused impact to email delivery for a subset of users
EX71628 – E-Mail and calendar access – Extended recovery
Jun 30, 2016 2:18 PM
Current Status
We’ve developed an additional fix to address the underlying cause of the issue. We’re preparing to deploy the fix to the affected environment to ensure that the issue does not reoccur.
User Impact
Users may be unable to send email messages through the Exchange Online service. Email messages may appear to be stuck in the Drafts or Outbox folders.
Scope of Impact
A few customers have reported this issue, and our analysis indicates that for most customers, it’s unlikely that many users would report impact related to this event.
- Start Time: Thursday, June 23, 2016, at 3:00 PM UTC
Preliminary Root Cause
An existing transport feature that is designed to expedite the delivery of email messages became degraded, which caused impact to email delivery for a subset of users.
Next Update by: Saturday, July 2, 2016, at 7:00 PM UTC
EX71674 – E-Mail timely delivery – Service restored
Jun 30, 2016 7:35 PM
Final Status
We’ve confirmed that the remaining message queues have now drained after implementing a configuration change to optimize message filtering.
User Impact
Users were experiencing delays when sending and receiving email messages. Affected users may have received Non-Delivery Reports (NDR) when sending email messages.
Scope of Impact
Customer reports indicated that many users likely experienced impact related to this event. Our analysis indicates that this issue may potentially have affected any of your users attempting to send or receive mail.
- Start Time: Thursday, June 30, 2016, at 2:30 PM UTC
- End Time: Thursday, June 30, 2016, at 11:30 PM UTC
Preliminary Root Cause
The infrastructure responsible for processing Exchange Online Protection (EOP) message filtering became degraded.
Next Steps
- We’re analyzing performance data and trends on the affected systems to help prevent this problem from happening again.
- We’re reviewing our code for optimizations and automated recovery options.
- We’ll publish a post-incident report within five business days.
EX71674 – E-Mail timely delivery – Service restored
Jul 1, 2016 12:08 AM
Final Status
We’ve rolled out the fix and confirmed that service is restored. Any meeting requests created during the outage will need to have the conference room calendar removed and readded to book the room.
User Impact
Scope of Impact
- Start Time: Monday, June 27, 2016, at 6:00 PM UTC
- End Time: Friday, July 1, 2016, at 2:54 AM UTC
Preliminary Root Cause
Next Steps
- We’re reviewing our deployment and provisioning procedures to help prevent this kind of problem in the future.
- We’ll publish a post-incident report within five business days.
Microsoft Office 365 is Microsoft’s new and popular way to license the Office products for online and desktop use. It also includes hosted Exchange for email, SharePoint, OneDrive for shared hard disk files, and the communications package Lync. The cloud based platform means Microsoft takes care of the system administration to update versions, apply security patches, monitor usage, ensure uptime and connectivity, and address hardware problems.
If you are still hosting your own Exchange Server in your facility, it’s time to consider outsourcing so Microsoft can worry about the versions, patches, hardware failures, and other maintenance chores. Microsoft will also host it in a real data center with reliable power sources, battery backups, multiple internet trunk lines, and enterprise quality physical security.
With Office 365, the hassles of hosting and maintaining your own SharePoint site is gone. Microsoft takes care of that for you and lets you create both private and public web sites.
Over the years, we’ve helped several clients better manage their emails and treat them like a database. We’ve built solutions that work with Exchange and Outlook to automatically classify contacts, tag emails, and store the information in a Microsoft SQL Server database. The data is presented through a Microsoft Outlook add-in showing all communications with a contact’s firm when you create or respond to an email. The data can also be displayed in the Facebook like interface to make it easy for everyone on your team to know what’s going on.