Microsoft Teams Application
Incident Report for CU Anschutz OIT | CU Denver OIT
Resolved
After extended monitoring and various optimizations and mitigation efforts, Microsoft has confirmed that its Microsoft Teams service and features have been restored or returned to optimal health.
Posted Jan 27, 2024 - 10:51 MST
Update
Microsoft is continuing to apply mitigations across the affected infrastructure and their telemetry is showing additional improvement in the user experience, though many customers are still affected by this issue. They’re also working to apply fixes to address individual affected Teams features in parallel while the broader remediation strategy is ongoing. Microsoft continues to evaluate any and all additional workstreams that will allow us to reduce the impact on customers that are still affected.

Next update: Monday 1/29/24 unless the impact increases significantly.
Posted Jan 26, 2024 - 16:16 MST
Update
Microsoft's network and backend service optimization efforts are ongoing and they’re monitoring internal telemetry to confirm that their mitigations are effectively reducing the impact to users. They continue to scale up services and identify new workstreams to remediate this problem more rapidly. Although many users remain impacted by this issue, they are seeing a reduction in errors and an increase in availability.
Posted Jan 26, 2024 - 16:11 MST
Update
Microsoft's failover operation did not provide the anticipated relief to end users
in North and South America regions, and they are now working to optimize traffic patterns as part of the mitigation effort. They're applying configuration changes across the affected network infrastructure to reduce customer impact as quickly as possible.
Posted Jan 26, 2024 - 13:34 MST
Identified
Microsoft has identified the issue and is implementing a fix.
Posted Jan 26, 2024 - 11:25 MST
Update
User impact: Users may experience multiple issues with their Microsoft Teams.

More info: Affected scenarios include, but aren't limited to:

- Users performing a cold boot may not able to log into teams and will see an "oops" page.
- Users logging in or unlocking their devices after some time may see missing messages.
- Users may fail to load messages in channels and chats
- Users are unable to view or download their media (images, videos, audio, call recordings, code snippets)
- Some messages may experience delays being sent
- Call Recordings might take longer to appear in user's OneDrive
- Bots may be unable to download attachments
- Sending and receiving read receipt notifications may be delayed

Current Status: As part of the failover process, Microsoft is currently scaling up the alternate infrastructure so that all the traffic can be distributed to the alternate infrastructure.
Posted Jan 26, 2024 - 11:00 MST
Investigating
Some users may experience errors when launching the Microsoft Teams web app. Microsoft is investigating a potential issue and checking for impact to multiple organizations. They will provide an update within one hour.
Posted Jan 26, 2024 - 10:39 MST
This incident affected: Other Systems and Services.