OIT Tech 32px

IT Outages & Scheduled Maintenance

Email list

Want to receive your own copy of all IT outage notices? Sign up for the IT Notifications List on Catmail.

Jump to Scheduled Maintenance...

Current Incidents

Adobe Connect 07/28/2016, 2:10am||Resolved 7:01 am

  • AFFECTED SERVICES:
     [Adobe Connect]
     
     
    IMPACT
      Intermittent interruptions

Titanium 07/28/2016 12:46am||Resolved at 7:01am

  • AFFECTED SERVICES:
     Titanium
     
    IMPACT
      Intermittent interruptions

OHIO ID Account Activation, down for some users, 07/24/16, 4:30 pm || RESTORED 07/24/16 9:45 pm

  • AFFECTED SERVICES:
     OHIO ID Account Activation
     
     
    IMPACT
      Down for some users
      
     
    Service has been restored 07/24/16 9:45 pm

Many OHIO online services, down for all users, 7/11/16, 10:45 pm || RESOLVED

  • 7/11/16, 11:52 pm: Services restored. Technicians continue to investigate the root cause.

    ----------------------------------------------------------------------------------------------

    AFFECTED SERVICES:
     Many university online services, including but not limited to:
     Oracle E-Business Suite & related apps
     PeopleSoft SIS, My OHIO Student Center & related apps
     
    IMPACT
      Down for all users

Southern campus network & telephones, down for all users, 07/08/16, 03:43 pm

  • AFFECTED SERVICES:
     Southern Campus - Storm related power outages
       Wired Internet
       WiFi
       University telephones
     
    IMPACT
      Down for all users
     
    Periodic updates will be posted until service has been restored.

Blackboard TurnItIn, down for all users, 7/8/16 6:30AM-10:15AM

  • AFFECTED SERVICE
      Blackboard Learn - TurnItIn module

    IMPACT
      Down for all users

    ROOT CAUSE
      Interruption in licensing. 
      
     

e-Business & related apps, degraded performance, 7/5/16, 8:00AM || RESOLVED

  • SERVICE RESTORED, 9:15AM, 7/5/16

    Oracle Forms functionality has been restored. 


    AFFECTED SERVICES:
     Oracle E-Business Suite: 
      Oracle HRMS
      Oracle FMS
      Employee Management System
      OUFMS Reporting
      My Personal Information
     BobcatBUY
     Online Graduate Appointments
     OU Funds Available
     OU PO Summary View
     
    IMPACT
      Degraded performance: users can log in to systems, but Oracle Forms are affected. 
      
    Periodic updates will be posted until service has been restored.

SCAN-to-EMAIL, down for all users, 6/30/16, 11:00AM || RESTORED

  • SERVICE RESTORED [7:35PM 6/30/16]:

    Please report any lingering issues to the IT Service Desk, 740-593-1222.

     

    See below for followup report from Microsoft:

    EX71674 - E-Mail timely delivery - Service restored
    Service restored - Jun 30, 2016 7:35 PM EDT

    Final Status: We've confirmed that the remaining message queues have now drained after implementing a configuration change to optimize message filtering.

    User Impact: Users were experiencing delays when sending and receiving email messages. Affected users may have received Non-Delivery Reports (NDR) when sending email messages.

    Scope of Impact: Customer reports indicated that many users likely experienced impact related to this event. Our analysis indicates that this issue may potentially have affected any of your users attempting to send or receive mail.

    Start Time: Thursday, June 30, 2016, at 2:30 PM UTC

    End Time: Thursday, June 30, 2016, at 11:30 PM UTC

    Preliminary Root Cause: The infrastructure responsible for processing Exchange Online Protection (EOP) message filtering became degraded.

    Next Steps: 
    - We're analyzing performance data and trends on the affected systems to help prevent this problem from happening again. 
    - We're reviewing our code for optimizations and automated recovery options.

    We'll publish a post-incident report within five business days.

     


    UPDATE [3:35PM, 6/30]:

     

    A service outage affecting many of Microsoft's Office 365 customers (including Ohio University) caused mail delivery delays. The problem has been resolved and backlogged mail should be delivered over the next several hours. 

    We will continue to monitor the issue here and with Microsoft. Due to the volume of users affected, the message backlog is extensive and may take time to clear. 

     


    AFFECTED SERVICES:
     Scan-to-Print functionality from Xerox MFPs (emails of scanned documents not arriving in recipient mailboxes)
     This appears to also affect SMS-to-Email messaging.
     
    IMPACT
      Down for all users
      
    Periodic updates will be posted until service has been restored.

Absence Management, down for all users, 6/30/16, 7:45am-9:48am || RESOLVED

  • RESTORED at 9:48am


    AFFECTED SERVICES:
     Absence Management

    IMPACT
      Down for all users

    Periodic updates will be posted until service has been restored.

Absence Management, degraded performance, 6/30/16, 5:05PM || RESTORED

  • SERVICE RESTORED 9:05AM, 7/1/16

    Please report any lingering issues to the OIT Service Desk at https://www.ohio.edu/oitech or 740-593-1222.

     


    AFFECTED SERVICES:

     

     Absence Management

    IMPACT
      Degraded performance

    Periodic updates will be posted until service has been restored.

EASTERN campus network & telephones, down for some users, 6/27/16, 8:37-9:50AM || RESOLVED

  • AFFECTED SERVICES:

    EASTERN Campus
       Wired Internet
       WiFi
       University telephones
       
    IMPACT
      Down for some users
      
    Issue was determined to have affected three users; formally resolved as of 9:50AM.

Network & telephones in HANING HALL, down for some users, 6/23/16, 10:55-11:05AM

  • AFFECTED SERVICES:

    HANING HALL
       Wired Internet
       WiFi
       University telephones
      
    IMPACT
      Down for some users
      
    Approximately 25 users experienced service interruption. Bad switch was qickly identified and replaced; issue is resolved. 

Network Attached Storage, degraded performance, 06/22/16, 03:50 pm || RESOLVED

  • 06/22/16, 04:30 pm: Service restored. Technicians continue to investigate the root cause.

    Please report any lingering issues to the OIT Service Desk at https://www.ohio.edu/oitech or 740-593-1222.

     

    AFFECTED SERVICES:
     Network Attached Storage
     
    IMPACT
      Degraded performance
     
    Periodic updates will be posted until service has been restored.

     

Ad Astra, down for all users, 6/14/16, 7:39 pm || RESOLVED

  • 6/15/16 7:04 pm - Service restored. The Ad Astra vendor was able to identify  and fix data integrity issues within  the application's database.  The vendor was unable to determine the cause, and will be analyzing application logs in search  of it.

    AFFECTED SERVICES

    • Ad Astra 

     

    IMPACT

    • Down for all users

     

    ADDITIONAL DETAILS

    • Earlier today, our vendor performed an upgrade. During post-upgrade functional testing, the application did not perform properly and has been down ever since. We have a support case open with the vendor but do not expect to see a resolution until at least the morning of 6/15.

e-Business & related apps, degraded performance, 06/08/16, 01:31 pm

  • AFFECTED SERVICES:
    •    Oracle E-Business Suite Workflow Mailers

    IMPACT:
    •    We are experiencing delays in Absence Management notifications and automated reports normally sent through Workflow Mailers
    o    Supervisors may continue to view or approve/reject absence requests in their Worklist located on the eBusiness Suite Home
    o    Page:  https://ebiz.ohio.edu/

Power outage affecting network on Ridges, West Green & Courtyard, down for all users 6/4/16, 9:40 am || RESOLVED

  • 6/4/16, 1:10 pm - Service restored.

    A power outage is affecting network & telephone services in some locations on the Athens campus. Facilities estimates restoration around 1:30 pm. Network will come back online shortly after power is restored.

     

    AFFECTED SERVICES

    Ridges, West Green & University Courtyard

     

    • Wired internet
    • WiFi
    • Telephones
    IMPACT
    • Down for all users in these locations

     

OHIO Front Door web site, degraded performance, 05/31/16, 9:56 am || RESOLVED

  • 5/31/16, 3:34 pm - Service restored. 


    AFFECTED SERVICES:
    •  Web pages hosted on www.ohio.edu
    •  OHIO Front Door website
    •  CommonSpot authoring
      
    IMPACT
    • Degraded performance : Web pages may return a "500" error or load partially with errors
    • Some CommonSpot authors will be unable to log in
    Periodic updates will be posted until service has been restored.

Box, down for some users, 5/31/16, 3:30 pm || RESOLVED

  • 5/31/16 5:30 pm - Service restored. Issue was on Box's end. 

    AFFECTED SERVICES:
     Box
      
    IMPACT
      Down for some users

    https://status.box.com/
      

Concur, degraded performance, 5/31/16, 3:01 pm

  • AFFECTED SERVICES:

    •  Concur

    IMPACT

    •   Degraded performance

    ADDITIONAL DETAILS

    • Note from Concur technical team: Please be advised that we are seeing reports of performance issues with both Travel and Expense this morning in the form of connection issues, slow performance, and problems completing travel transactions. Please continue to log cases as you normally would and I will escalate them accordingly. Our development teams are actively working to resolve any issues that clients are facing.
    • 3:10 pm follow up from Concur: Though Expense and Travel have regained login capabilities, many of you are still experiencing slow performance from the first P-1 we had this morning. As you are now able to access the Support portal, please continue to log performance cases as you normally would. My apologies for any inconvenience this causes you and your users; please know that our development teams continue to work tirelessly to investigate and ensure our systems go back to working as normal.

OBI Reporting, down for all users, 05/26/16, 07:38 am/pm || RESOLVED

  • 5/26/16 7:44 am - Issue was an unintended side effect of automated patching. Will explore options to prevent this problem in the future.


    Service restored AFFECTED SERVICES:
      Oracle Business Intelligence (OBI Reporting)
     
    IMPACT
      Down for all users
     

    05/26/16, 07:44 am/pm: Service restored.

    Please report any lingering issues to the OIT Service Desk at https://www.ohio.edu/oitech or 740-593-1222.

Footprints, down for all users, 5/18/16, 6:00 am - 7:10 am || RESOLVED

  • 5/18/16 - Service restored after installing an updated security certificate in the app.

    AFFECTED SERVICES

    • Footprints

    IMPACT

    • Down for all users

Identity Manager (ITIM), down for all users, 5/16/16 3:28 pm || RESOLVED

  • 5/16/16, 5:25 pm - Service restored. Outage caused by a server process running out of memory. We will be investigating why this particular process is having this issue.


    AFFECTED SERVICES

    • Identity Manager (ITIM)
    • Password resets, account activations
    IMPACT
    • Down for all users
    •  

     

Most OHIO online services, down for all users, 5/15/16, 2:15 am || RESOLVED

  • 5/15/16 5:42 am - Service restored. Outage appears to be related to other recent firewall problems. We will continue to work with our vendor to identify a permanent fix.


    AFFECTED SERVICES:
     Most university online services, including but not limited to:
     Blackboard Learn
     Catmail
     Oracle E-Business Suite & related apps
     PeopleSoft SIS, My OHIO Student Center & related apps
      
    IMPACT
      Down for all users

Most OHIO online services, down for all users, 5/13/16, 8:17 am || RESOLVED

  • 5/13/16, 9:45 am - Service restored after rebooting the university’s firewall. Technicians continue to work with engineers from our firewall’s manufacturer to determine whether this outage is related to similar outages in the past and to identify steps to prevent future outages.


    AFFECTED SERVICES:
     Most university online services, including but not limited to:
     Blackboard Learn
     Catmail
     Oracle E-Business Suite & related apps
     PeopleSoft SIS, My OHIO Student Center & related apps
      
    IMPACT
      Down for all users
      
    Periodic updates will be posted until service has been restored.

NetReg, down for all users, 5/13/16 11:06 am - 11:17 am || RESOLVED

  • Service restored after restarting the app. Likely a side effect of the earlier data center network outage.

    AFFECTED SERVICES

    • NetReg

    IMPACT

    • Down for all users

Catmail. service advisory from Microsoft, 5/13/16

  • *Incident EX46155 - E-Mail and calendar access - Service degradation*
    *Update:* May 13, 2016 15:43 PM EDT

    *Current Status:* Engineers are investigating an issue in which some users are experiencing problems with meeting update notifications on their iOS and Android mobile devices. It was determined that a recent update caused this issue. Engineers have developed a fix for the underlying code issue and are now testing it. Engineers anticipate that the solution will be ready to begin deployment within 4-6 hours. Once the deployment is underway, it may take up to 72 hours to complete.

    *User Impact:* Users are receiving repeated meeting update notifications on their iOS, Android, and Windows Mobile devices. This is only affecting meetings that were created or updated from an iOS, Android, or Windows Mobile device connecting to the service via Exchange ActiveSync (EAS).

    *Scope of Impact:* A very limited number of customers and users appear to be impacted. Some customers have reported that their organization is affected by this issue.

    *Incident Start Time:* Wednesday, May 4, 2016, at 7:00 PM UTC

    *Preliminary Root Cause:* An update intended to enhance Exchange Online service functionality caused a code regression within the Exchange ActiveSync component that processes meeting notifications.

    *Next Update by:* Saturday, May 14, 2016, at 1:00 AM UTC

Account activations & password changes, down for all users 5/12/16, 1:27 pm || RESOLVED

  • 5/12/16, 2:30 pm: Service restored. Outage cause: canceling of an identity feed triggered a process that resulted in a database lock 

     

     

    AFFECTED SERVICES

    • iTIM
    • Account activations
    • Password changes

    IMPACT

    • Down for all users
     

Network & telephones in Wilson Hall, degraded performance, 5/9/16, 10:17 am

  • AFFECTED SERVICES:
    Wilson Hall (College Green)
       Wired Internet
       WiFi
       University telephones
      
    IMPACT
      Degraded performance and intermittent outages in some areas.
      
    Periodic updates will be posted until service has been restored.

     

Network & telephones, degraded performance, 5/9/16, 8:57 am

  • 5/9/16 - 8:06 pm - 8:24 pm Core routers rebooted. Network down for all users during this reboot.


    AFFECTED SERVICES

    • Wired internet
    • WiFi
    • Telephones

    IMPACT

    • Intermittent degraded performance, including slow network access and telephone calls where one party is unable to hear the other.

    ADDITIONAL DETAILS

    • We have received reports of this issue in Baker, Chubb, Cutler, Clippinger, Grover, Stocker and Wilson Hall (administrative building). If you encounter these issues in your location, please report them to the IT Service Desk, https://www.ohio.edu/oitech or 740-593-1222.

University telephones in Chubb Hall, down for some users, 5/9/16, 8:57am

  • Update, 11:35am:

    Technicians are working with our vendor in continued investigation of telephone network issues. 

    Please be advised that some computers' wired connections may be affected by the same problem affecting the phones; we are looking into the entirety of network reports in the building. 


    AFFECTED SERVICES:
      University telephones in Chubb Hall
      
    IMPACT
      Down for some users
      
    Periodic updates will be posted until service has been restored.

Proctorville, Cambridge & Zanesville network, down for all users, 5/9/16 12:02 am - 7:06 am || RESOLVED

  • Service restored for Cambridge & Zanesville at 12:52 am, Proctorville at 7:06am. Issue was related to an Internet provider equipment failure.

    AFFECTED SERVICES

    Proctorville, Cambridge & Zanesville network

    IMPACT

    Down for all users

Network & telephones in Baker Center, degraded performance, 5/9/16, 9:53 am

  • AFFECTED SERVICES:
    Baker University Center
       Wired Internet
       WiFi
       University telephones
      
    IMPACT
      Degraded performance and intermittent outages in some areas.
      
    Periodic updates will be posted until service has been restored.

Adobe Connect, degraded performance, 5/6/16, 10:51 pm

  • 5/12/16 - Service currently is available; however, a few student account logins may not work correctly. If you are unable to invite a student to a meeting room, set the meeting to either open access or to allow approved guest access and have that student enter as a guest.


    AFFECTED SERVICES

    Adobe Connect

    IMPACT

    degraded performance - users unable to create new meetings

Network and telephone service, down for some users, 5/6/16, 1:40pm || RESOLVED

  • 5/6/16, 9:50 pm - Service restored. 

    Outage was caused by physical damage to a major fiber optic cable. The cable was not entirely severed, which is why the issue was only intermittent. We have bypassed the section until repairs can be completed. In the meantime, network performance should be back to normal. 

    If you notice any lingering issues, please report them to the IT Service Desk at https://www.ohio.edu/oitech or 740-593-1222.


    5/6/16, 8:01 pm - Techs continue to work with vendor to troubleshoot intermittent issues. Will update status in morning, or sooner if issue is resolved.


    5/6/16, 4:15 pm: We will be disabling and re-enabling the network in individual buildings in an attempt to isolate the issue. 


    5/6/16, 3:50 pm: Network and telephones continue to experience intermittent interruptions for some users. Vendor support engineers are assisting with the troubleshooting.


    AFFECTED SERVICES:
     Wired Internet
     WiFi
     University telephones
       
    IMPACT
      Down for some users
      
    Periodic updates will be posted until service has been restored.

e-Business & related apps, down for all users, 05/04/16, 08:31-8:54 am || RESTORED

  •  

    8:54 am am/pm: Service restored. 

    Please report any lingering issues to the OIT Service Desk at https://www.ohio.edu/oitech or 740-593-1222.


    AFFECTED SERVICES:
     Oracle E-Business Suite:
      Oracle HRMS
      Oracle FMS
      Employee Management System
      OUFMS Reporting
      My Personal Information
     Online Graduate Appointments
     OU Funds Available
     OU PO Summary View
     
    IMPACT
      Down for all users
     
    Periodic updates will be posted until service has been restored.

e-Biz & My HR, intermittent interruptions, 5/3/16, 6:11 pm || RESOLVED

  • 6:36 pm: Service restored. We will continue to monitor the system as usage picks back up tomorrow morning.

     

     

    AFFECTED SERVICES:
     Oracle E-Business Suite: 
      Oracle HRMS
      Oracle FMS
      Employee Management System
      OUFMS Reporting
      My Personal Information
     
    IMPACT

    • Intermittent login interruptions
    • System will be rebooted to restore service. This could cause a brief outage.
     

MyHR (My Personal Information), down for all users, 5/2/16. 7:43 - 8:36 am || RESOLVED

  • Service restored at 8:36 am.
     


    AFFECTED SERVICES:
     My Personal Information (myhr.ohio.edu)
     
    IMPACT
      Down for all users
      
    Periodic updates will be posted until service has been restored.
      

Pharos printing, down for all users, 4/28/16, 1:34-2:28AM || RESOLVED

  •  

    AFFECTED SERVICES:
      Student printing via Pharos
      Faculty/Staff printing via Pharos
        
    IMPACT
      Down for all users
        
    Service restored after restarting a system that did not come back online after an automated system update.

     

MyOHIO portal Catmail preview, down for all users, 4/25/16, 9:13 am || RESOLVED

  • 9:57 am - Service restored after restarting a process that had stopped responding.


    AFFECTED SERVICES

    • Catmail preview tool inside the MyOHIO portal
    IMPACT
    • Down for all users
    Currently restarting the affected service - should restore service within 30 minutes. 
     
    Catmail itself is not down, just the portal preview. Catmail can be reached directly at https://catmail.ohio.edu

     

Bobcay Buy, down for all users, 4/21/16, 9:55AM || RESOLVED

  • 11:02 AM Service restored.

    Please report any lingering issues to the OIT Service Desk at https://www.ohio.edu/oitech or 740-593-1222.


    AFFECTED SERVICES:
     Bobcat Buy

    IMPACT
      Down for all users

    Periodic updates will be posted until service has been restored.

Network & telephones in Tiffin, down for some users, 04/19/16, 08:36 am || RESOLVED

  • 04/19/16, 09:23 am: Service restored.

    Please report any lingering issues to the OIT Service Desk at https://www.ohio.edu/oitech or 740-593-1222.

     

    AFFECTED SERVICES:
     Tiffin
       Wired Internet
       WiFi
       University telephones
     
    IMPACT
      Down for some users
     
    Periodic updates will be posted until service has been restored.

Dublin Campus power outage may affect network & telephones, 04/19/16, 12:45-1:34pm || RESTORED

  • Electrical service restored per Dublin campus and AEP (approx. 1:30PM).


    There is currently a power outage on the Dublin campus, affecting buildings A and D. AEP advises this was caused by a substation failure, and are projecting restoration of power as early as 3:00pm today.

    Network battery backups report sufficient supply for at least the next few hours, but should they be exhausted, both network and telephone connectivity will be affected.

     

     

Front Door web pages, down for some users, 4/19/16 11:21 am || RESOLVED

  • 4/19/16, 11:33 am - Service restored after restarting the affected server.


    AFFECTED SERVICES

    • Front Door web pages hosted on ww2 (static page server)

    IMPACT

    • Down for some users

Class Climate, degraded performance, 4/18/16, 11:13am || RESOLVED

  • 4/19/16 12:00 pm - Service restored. Slowness was caused by high traffic volumes (over 90 surveys active at the same time).


    AFFECTED SERVICES
      Class Climate
      
    IMPACT
      Degraded performance
      

OneCard/eAccounts intermittent interruptions 4/15/16, 11:15AM || RESOLVED

  • 4/15/16, 3:39 pm - Service restored. Outage was caused by a problem on the vendor's side that affected all of their uinversity customers across the country.


    AFFECTED SERVICES:
     OneCard / eAccounts
     
    IMPACT
      Intermittent interruptions to OneCard / eAccounts; users may be unable to access OneCard website. 
      
    Periodic updates will be posted until service has been restored.

Class Climate, down for some users, 4/15/16 1:57 pm || RESOLVED

  •  

    4/21/16 - The Class Climate email issues we had on Friday April 15th were caused by human error when someone accidentally deleted data from the system. We have rebuilt the lost participant emails to the best of our abilities for the current term. To reduce the likelihood of this happening again, we will be contacting everyone with administrative rights to make sure they understand how this happened and what they should do to avoid it in the future. We also will investigate technical options to restrict access to this particular function.


    4/15/16, 3:00 pm - Service restored. Email addresses were added back into the system.


     

    AFFECTED SERVICES

    • Class Climate

    IMPACT

    • Down for some users
    • Unable to distribute evaluations because email addresses come back as missing
     

     

Network in Academic Research Center, down for some users, 04/15/16, 08:38 am RESOLVED

  • 04/15/16, 09:06 am: Service restored.

    Please report any lingering issues to the OIT Service Desk at https://www.ohio.edu/oitech or 740-593-1222.

     

    AFFECTED SERVICES:
     Academic Research Center
       3rd floor wired internet / telephones and wifi coverage on 3rd floor
         
     
    IMPACT
      Down for some users
     
    Service was down for approx 10 minutes, caused by faulty AC unit, Facilities staff currently on site for repair.
     

Lancaster campus network & telephones, down for all users, 04/15/16, 09:34 am || RESOLVED

  • 4/15/16, 4:21 pm - Service restored. Lancaster campus Facilities has confirmed that he electrical power problems that caused this outage have been repaired.


    2:55 pm, 4/15/16 - This outage was the result of a partial power outage that is affecting our networking equipment on the Lancaster campus. Power still has not been fully restored, so the outage continues.


    AFFECTED SERVICES:
     Lancaster Campus
       Wired Internet
       WiFi
       University telephones
     
    IMPACT
      Down for all users
     
    Periodic updates will be posted until service has been restored.

Pharos printing, 4/15/16, 3:07 pm || RESOLVED

  • 4/15/16, 5:03 pm - Service restored. Technicians continue to investigate the root cause of the problem. Please report any lingering issues to the OIT Service Desk at 740-593-1222 or https://www.ohio.edu/oitech 

    AFFECTED SERVICES

    • Student printing via Pharos
    • Faculty/staff printing via Pharos

    IMPACT

    • Degraded performance

Network & telephones at the Ridges and University Courtyard, down for most users, 04/13/16, 2:15PM-2:55PM || RESOLVED

  •  

    RESOLVED, 2:55PM. Technicians determined root cause was a damaged fiber. 


    2:46pm - Users continue to report issues. Technicians are investigating. 


    04/13/16, 2:28pm: Service restored. Technicians continue to investigate the root cause.

    Please report any lingering issues to the OIT Service Desk at https://www.ohio.edu/oitech or 740-593-1222.


    AFFECTED SERVICES:
    The Ridges and University Courtyard
       Wired Internet
       WiFi
       University telephones
      
    IMPACT
      Down for most users
      
    Periodic updates will be posted until service has been restored.

Kodak Prinergy server - degraded performance 4/5/16, 8:12am

  • AFFECTED SERVICES:
     Kodak Prinergy server, performance degraded. File transfers very slow. 
     
    IMPACT
      Degraded performance
      
    Periodic updates will be posted until service has been restored.

Mac Mail & iCal, down for some users, 04/05/16, 04:13 pm || RESOLVED

  • 4/11/16, 3:25 pm: Service restored. Follow up from Microsoft:

    *Exchange Online Incident EX44931*

    *Final Status:* The deployment of the fix is complete and engineers have confirmed service restoration.

    *User Impact:* Affected users were unable to synchronize their Exchange Online calendars when connecting via EWS (Exchange Web Services). This issue only affected calendar synchronization between Exchange Online and the email app on Mac OS X.

    *Scope of Impact:* Many customers likely had users experiencing impact related to this event. Some customers reported that their organization was affected by this issue.

    *Incident Start Time:* Tuesday, March 29, 2016, at 10:47 PM UTC

    *Incident End Time:* Monday, April 11, 2016, at 6:00 PM UTC

    *Preliminary Root Cause:* A recent update was deployed that caused a configuration issue affecting calendar synchronization via EWS.

    *Next Steps:* Review procedures for validating service updates as the configuration issue was not identified during the testing phase of the development process. A post-incident report will be available on the Service Health Dashboard within five business days.


    AFFECTED SERVICES
      Mac Mail & iCal
     
    IMPACT
      Down for some users

    ADDITIONAL DETAILS

    Microsoft has identified an issue on their end that is affecting calendar synchronization for some Mac iCal users. They will be rolling out a fix in the next few days. It's unclear if this fix will also resolve similar sync problems with Mac Mail. Outlook and OWA are not affected by this issue.
     
    Periodic updates will be posted until service has been restored.

Bobcat Buy, down for all users, 4/1/16, 11:15AM || RESOLVED

  • 4/1/16, 1:17 PM: Service restored.

    Please report any lingering issues to the OIT Service Desk at https://www.ohio.edu/oitech or 740-593-1222.


    AFFECTED SERVICES:
     Bobcat Buy

    IMPACT
      Down for all users

    Periodic updates will be posted until service has been restored.

DUBLIN campus network & telephones, degraded performance, 3/28/16, 10:15 am || RESOLVED

  • 3/28/16, 3:45pm: Service restored. 

    Please report any lingering issues to the OIT Service Desk at https://www.ohio.edu/oitech or 740-593-1222.


    CORRECTION: The UPS replacement will occur shortly, and is expected to be complete by 3:30PM.


    UPDATE: Technicians are replacing the UPS for the POS firewall and Pharos/SCCM server. There will be a brief service interruption between 12:45-2:00. 


     

    AFFECTED SERVICES:
      DUBLIN Campus
       Wired Internet
       WiFi
       University telephones
       
    IMPACT
      Intermittent interruptions; appear to be due to power failure which occurred shortly after 10:00am.

    Routers are back up, but some firewalls are still down, and the voicegateway is nonresponsive.
      
    Periodic updates will be posted until service has been restored.
      

PeopleSoft SIS & OBI Reporting: down for all users, 3/23/16, 8:25 pm || RESOLVED

  • 3/24/16, 2:20 am - OBI service restored. App has temporarily been moved out from behind SSO while we identify the root cause of the problem.


    9:11 pm - OBI reporting remains unavailable. Techs investigating. All other services working normally.


    8:49 pm - Service mostly restored. OBI reporting may take a few minutes to resume accepting logins


    Outage caused by an expired security certificate. Service should be restored shortly.

    AFFECTED SERVICES

    • PeopleSoft SIS
    • OBI reporting

    IMPACT

    • users unable to log in

OnBase, degraded performance, 3/19/16 11:19 am || RESTORED

  • 3/19/16, 12:32 pm - service restored. Outage caused by expired security certificate

    AFFECTED SERVICES

    • OnBase

    IMPACT

    • Degraded performance: users receiving an 'untrusted connection' when trying to access the app

Pharos printing & VDI/VMware Horizon View, 3/19/16, 9:07 am

  • 3/19/16, 11:08 am - Service restored

    Additional work will be needed during tonight's maintenance window to ensure that issues do not come back. Our vendor engineers have yet to isolate the root cause of these network problems, so work likely will be needed during future maintenance windows as well.


    3/19/16, 10:25 am - We have made some network changes that appear to have eliminated the issues. Please report any failed print jobs or VDI session interruptions to the OIT Service Desk.


    AFFECTED SERVICES

    • Pharos printing 
      • Student printing in Athens campus locations like Alden Library
      • Faculty/staff printing via Pharos
    • VDI/VMware Horizon View client

     

    IMPACT

    • Degraded performance
    • Student print jobs may fail and result in a "[free print]" error. 
    • Faculty/staff print jobs will fail silently with no error message
    • Possibility of interrupted connections for VDI sessions

Concur, down for all users, 03/18/16, 10:34 am || RESTORED

  • 3/19/16, 1:45 pm - Service restored


    3/19/16, 9:40 am - Concur is still down this morning. Unclear whether the vendor applied their fix last night or not. We have reported the issue to them.


    3/18/16, 9:40 pm - Concur remains down, but their technical support states that they have made progress in identifying a fix and intend to apply that fix later tonight.


    3/18/16, 12:24 pm - We have received word from our Concur vendor that this a global problem on their end affecting all Concur users. They expect to have the problem resolved before the close of business today. We will send an update when service is restored.


    03/18/16, 10:34 am: Technicians continue to troubleshoot. We will update this status every two hours until the outage has been resolved.


    AFFECTED SERVICES:
     Concur
     
    IMPACT
      Down for all users
     
    Periodic updates will be posted until service has been restored.

e-Business & related apps, degraded performance, 03/17/16, 5:10 pm || RESOLVED

  • 03/17/16, 05:10pm: Service restored. Technicians continue to investigate the root cause.

    Please report any lingering issues to the OIT Service Desk at https://www.ohio.edu/oitech or 740-593-1222.

     

    AFFECTED SERVICES:

     Oracle E-Business Suite:

      Oracle HRMS

      Oracle FMS

      Employee Management System

      OUFMS Reporting

      My Personal Information

     BobcatBUY

     Online Graduate Appointments

     OU Funds Available

     OU PO Summary View

     

    IMPACT

      Degraded performance

     

    Periodic updates will be posted until service has been restored.

PeopleSoft SIS & related apps, down for some users, 3/16/16, 2:00 pm - 2:24 pm || RESOLVED

  •  

    03/16/16, 2:24 pm: Service restored. Technicians continue to investigate the root cause.

    Please report any lingering issues to the OIT Service Desk at https://www.ohio.edu/oitech or 740-593-1222.

    ___________________________________________________________________________________________

     

    AFFECTED SERVICES:
      Faculty & Advising Center
        Advisee lists
        Class lists
        DARS
        Grade rosters
        Textbook
    PeopleSoft administrative screens
    Authorized User payment portal
    Enrollment and degree verification letters
      
    IMPACT
      Down for some users
      
    Periodic updates will be posted until service has been restored.

HIPAA virtual desktop environment, down for some users, 3/14/16, 3:52 pm - 6:25 pm || RESOLVED

  • Service restored at 6:25 pm. Outage caused when devices lost their DHCP connections.

    AFFECTED SERVICES

    • HIPAA virtual desktop (VDI) environment
    IMPACT
    • Down for some users

Alpha Xi Delta wifi, down for some users, 3/12/16, 7:30 am || RESTORED

  • 3/12/16, 8:18 am - Service restored. Outage was caused by a faulty UPS unit.

     

    AFFECTED SERVICES

    • Wireless Internet in Alpha Xi Delta

     

    IMPACT

    • Down for some users

Pharos printing & VDI/VMware Horizon View, 3/11/16, 7:59 am

  • 3/13/16, 11:57 pm - Service should not be experiencing any network errors at this point; however, additional work still may be needed in the future.

    3/11, 6:26 pm - We will be making some changes tonight to our central network equipment. Expect a single, less than 5 minute outage some time during the published 12:05 am - 2:00 am window on 3/12. Engineers will continue to troubleshoot throughout the weekend, but with no expected outages other than those during the already published nightly window.

    Friday, 3/11/16, 4:30 pm

    Changes made during today's troubleshooting have temporarily resolved the network issues that were affecting Pharos printing and VDI/VMware Horizon View; however, the problem has only been bypassed. Additional work will be needed to identify the root cause and apply a permanent fix.

    This is a complicated process: Multiple engineers from our network vendor have been on site working in tandem with OIT staff since March 10. The troubleshooting has involved multiple changes to our network setup followed by intense analysis of the effects of those changes. For example, today's work involved replicating the conditions of last month's outages and then capturing data from several interruptions as they happened. 

    This detailed analysis is starting to pay off. We have identified likely problem areas and will focus our attention on those areas in the coming days. Doing so may require some brief interruptions during our advertised 12:05 am - 2:00 am windows. We will do our best to minimize any such interruptions.


     

    AFFECTED SERVICES:

    •   ID card swipe student printing
    •   Faculty/Staff printing via Pharos
    •   VDI/VMware Horizon View  

     

    IMPACT

    •   Intermittent interruptions

      
    These interruptions are part of ongoing work to identify and repair the problem that caused outages during February.

Regional campus network & telephones, degraded performance, 03/09/16, 4:33 pm || RESOLVED

  • Wednesday, March 09, 2016; 4:33 pm

     

    AFFECTED SERVICES:

      Proctorville Center

       Wired Internet

       WiFi

       University telephones

     

    IMPACT

     

    At approximately 4:30pm , the Proctorville Center experienced a brief disruption to network services, which has been resolved.

    Those experiencing continued difficulties are asked to contact the Service Desk at 740-593-1222

DARS, down for all users, 3/3/16, 11:10 am || RESOLVED

  • 3/3/16, 12:07 pm - Service restored. Additional system resources have been allocated to the affected server to prevent a recurrence of this outage.

     

     

    AFFECTED SERVICES

    • DARS

     

    IMPACT

    • Down for all users

     

Advance alumni app, down for all users, 3/2/16, 11:15 am - 11:59 am || RESOLVED

  • Service restored. Techs still investigating cause.

     

    AFFECTED SERVICES

     

    Advance alumni app

     

    IMPACT

     

    down for all users

CommonSpot authoring, down for all users, 2/29/16, 8:07 am || RESOLVED

  • 2/29/16, 9:52 am - Service restored. Outage caused by a server that did not restart properly after an automated system update.


    AFFECTED SERVICES

     CommonSpot authoring

     

    IMPACT

      Down for all users

University printers, intermittent interruptions, 2/25/16, 10:37 am || RESOLVED

  • Monday, 2/29/16, 4:46 pm - Following yesterday's data center maintenance, most of the ID swipe and Pharos printing errors appear to have been resolved; however, we will continue to keep a close eye on the service as students return from spring break. 

    If you continue to have problems with documents not printing, please report them to your local lab manager or printer support staff. You also can call the OIT Service Desk at 740-593-1222 or submit a request online at https://www.ohio.edu/oitech


    Friday, 2/26/16, 1:38 pm

    Technicians continue to troubleshoot intermittent interruptions with student ID-swipe printers and faculty/staff Pharos printing.

    This Sunday at 12:05 am, scheduled maintenance in the university data center that may help resolve the problem. In the absence of any new developments, the next update on this outage will be sent on Sunday.


    2/25/16, 11:10 am - Technicians continue to troubleshoot. Until this issue is resolved, you usually can fix a failed print job with the following steps:

    Students: Print job fails with an error message.   

    1. Go to a different ID-swipe printer within 1 hour.
    2. Swipe your ID card and look in your "hold queue" for a print job with the label "[Free Print]".
    3. Select that job, and your document should print.

     

    Faculty/Staff: Print job fails to print, no error message.

    1. Reboot the affected printer.
    2. Re-send the document that failed to print.

    AFFECTED SERVICES:

    •   ID card swipe student printing
    •   Faculty/Staff printing via Pharos

      
    IMPACT

    • Intermittent interruptions
    • Athens campus printers only

     

    ADDITIONAL DETAILS

    • Print jobs will fail randomly, often with one of the following errors:
    • Error releasing print job: Cannot connect to printer
    • Error releasing print job: Failed to render document
    • In some cases, students may be charged for the failed job
    • Restarting the affected printer will sometimes allow the job to complete
    • Vendor technical support is involved in troubleshooting this problem

Alumni, down for all users, 2/29/16, 10:18 am || RESOLVED

  • 2/29/16 10:45 am: Service restored. Technicians continue to investigate cause.


    AFFECTED SERVICES

      Alumni web app

    IMPACT

      Down for all users

Network & telephones in ARC, down for all users, 2/29/16, 4:17 am || RESOLVED

  • 2/29/16, 5:15 am: Service restored after resetting a tripped electrical breaker.

     

    AFFECTED SERVICES

    Academic Research Center (ARC)

      Wired Internet

      WiFi

      Telephones

     

    IMPACT

      Down for some users on 2nd floor

Alden Library network, down for some users, 2/26/16, 9:57 am || RESOLVED

  • 2/26/16, 12:10 pm: Network switch has been replaced. Service restored.


    2/26/16, 11:08 am:

    Alden locations where users may experience streaming issues: 301a through m, 301t, 303, 308,309,310,312,319,322,323,325,327 and 330

    Network tech will attempt to replace the failing piece of hardware in the next hour but will adjust timing of the repair if needed to avoid disrupting any events happening in those rooms.


    2/26/16, 10:23 am: Service partially restored. Users able to connect but may have trouble streaming. Tech will replace malfunctioning 3rd floor network switch asap.


    AFFECTED SERVICES

    Alden Library

    • Wired internet
    • WiFi
    IMPACT
    • Down for some users
    • Technician on way to site to reboot malfunctioning piece of hardware. 
    •  

OULN TMS server, down for all users, 2/26/16 10:00 am || RESOLVED

  • Service restored

    AFFECTED SERVICES

    • OULN TMS server

     

    IMPACT

    • Down for all users

VDI, down for some users, 2/26/16, 6:56 am || RESOLVED

  • 2:54 pm - Service restored


    AFFECTED SERVICES

    • VDI
    • VMwarwe Horizon View

     

    IMPACT

    • Down for some users
    • Client returns a black screen after logging in

OHIO Front Door web server, 2/25/16, 7:29 am || RESOLVED

  • 8:20 am - We are temporarily disabling a server that did not come back up properly after an automatic system update. This should eliminate the 'server not responding' errors.


    AFFECTED SERVICES

    • OHIO Front Door
    • Some pages hosted on www.ohio.edu may be unavailable

    IMPACT

    • Intermittent interruptions

ColdFusion/OHIO Front Door, intermittent interruptions for some users, 2/24/16, 7:57 am || RESOLVED

  • 8:24 am - Issue appears to have been related to an automated system update.

    2/24/16, 8:06 am - Service restored. Currently working with server admin to identify cause.

    AFFECTED SERVICES

    • Cold Fusion server (OHIO Front Door)
    IMPACT
    • Intermittent interruptions for some users (very small subset of CommonSpot users)
    •  

Multiple academic & admin apps, OHIO Front Door, down for all users, 2/23/16, 5:21 am - 9:50 am || RESOLVED

  • 9:52 am - Outage was caused by a hardware failure in a piece of data center infrastructure combined with a bug that prevented secondary/backup hardware from taking over seamlessly. Defective unit has been replaced, and our vendor will be sending us replacements for the backup units as well that have been patched to remove the bug that caused the problem.


    8:34 am - DARS service restored. Techs continue to investigate


    8:30 am - DARS has stopped processing requests. Technicians are restarting the server.


    Service restored. Technicians still investigating root cause.

     

     AFFECTED SERVICES

     

    • AdAstra
    • Apex
    • Custom apps (PeopleSoft)
    • DARS
    • Mulesoft
    • NetOhio
    • NetReg
    • OBI scheduler
    • OHIO Front Door web server (CommonSpot)
    • VMware Vcenter
    • Workforce 
    • Alumni Services - Ellucian Advance

    IMPACT

     

    • Down for all users

Funds Available View, down for all users, 2/22/16, 8:54 am || RESOLVED

  • Service restored

    AFFECTED SERVICES

    • Funds Available View
    IMPACT
    • Down for all users

e-biz & related apps, down for all users, 2/22/16 7:58am - 8:15 am || RESOLVED

  • 8:15 am - Service restored after server restart

    AFFECTED SERVICES

    • e-Business & related apps

     

    IMPACT

    • down for all users

OULN TMS server: 2/22/16, 3 pm - 3:34 pm || RESOLVED

  • Outage was an issue with the connection between the server and remote locations. Server recovered on its own.

    AFFECTED SERVICES

    • OULN TMS server

     

    IMPACT

    • down for all users

Feb 12, 8:49 am: Pharos printing, intermittent interruptions || RESOLVED

  • 2/25/16, 10:07 am - We continue to see random print job failures for both student ID card swipe printing and faculty/staff printing via Pharos. IT technicians and vendor tech support continue to look for a solution.


    2/22/16, 6:45 pm - OIT and vendor technicians spent today troubleshooting intensively on the Pharos print environment; however, the service continues to experience intermittent interruptions that require restarting individual printers to allow print jobs to complete. We will post an update again tomorrow.


    2/21/16, 4:55 pm - Alden Library experienced a period of disruption where printers needed to be restarted before nearly every print job. After several server restarts, service appears to be working normally again. We continue to work with our vendor to identify the cause of this problem.


    2/19/16, 2:41 pm - Printers continue to experience occasional interruptions. Restarting the affected printer will restore service. Vendor technical support and OIT staff continue to work on indentifying and eliminating the cause.


    AFFECTED SERVICES

        Pharos printing for faculty/staff
        Student ID card swipe printing in Alden Library and other locations

    IMPACT

        Intermittent interruptions

Business systems web apps, 2/22/16, 8:51 am - 9:46 am || RESTORED

  • Service restored after restarting a server that did not come back online properly after an automated system update

     

    AFFECTED SERVICES

    • Business systems web apps, including FA View and Student Rapid Hire
    IMPACT
    • down for some users

Feb 20, 8:15 am - 8:50 am: Multiple acad & admin apps || RESTORED

  • Outage was caused by a server not properly coming back online after a system update.

     

    AFFECTED SERVICES

     

    • PeopleSoft custom apps, including Faculty & Advising Center
    • NetReg
    IMPACT
    • Down for all users

Feb 20, 9:23 am - 9:56 am: Pickering Hall Internet, down for some users || RESTORED

  • Outage was caused by a failed power supply in a piece of networking equipment. Service restored after replacing the faulty part

     

    AFFECTED SERVICES

    • Wired & wireless Internet in Pickering Hall
    IMPACT
    • Down for some users

Feb 19, 2:14 am - 2:49 am: Clippinger Internet & telephones || RESOLVED

  • Friday, 2/19/16, 2:14 am - 2:49 am

     

    Service was restored at 2:49 am when technician replaced a failed power supply for a piece of networking equipment

     

    AFFECTED SERVICES

    • Clippinger Hall Internet & telephones
    IMPACT
    • Down for some users

Feb 18, 8:31 am: DCVPN || RESOLVED

  • 10:33 am - Outage appears to have been caused by a transient issue and should not recur. We will continue to monitor the affected server.

    8:40 am - Service restored after temporarily disabling a malfunctioning server. We currently are investigating the root cause of the problem.

     

     

    AFFECTED SERVICES

    • DCVPN
    IMPACT
    • Users unable to log in to the service
    • Impact limited to SIS functional offices and IT developers

FEb 18, 9:45 am: VMware Horizon View (VDI), down for some users || RESOLVED

  • AFFECTED SERVICES

    • VMware Horizon View (VDI)
    IMPACT
    • Service down for some users
    • Vendor tech support currently is investigating

Feb 17, 4:55 pm - 5:32 pm: VDI || RESOLVED

  • 5:32 pm -- Service restored

    AFFECTED SERVICES

    • VDI

     

    IMPACT

    • Users were unable to start new VDI sessions for approximately 20 minutes
    • Brief blip at 5:15pm that woudl have interrupted all existing sessions, but connections restored cleanly

Feb 17, 9:55 am - 10:24 am: Alumni Advancement || RESOLVED

  • Service restored after restarting a malfunctioning web service.

    AFFECTED SERVICES

    • Alumni Advancement

     

    IMPACT

    • Down for all users

Feb 17, 8:22 am: OnBase Unity client, intermittent interruptions

  • 2/19: 9:03 am: Service restored, root cause unknown. Please report continuing issues to Service Desk.

    2/19, 8:37 am: Users reporting login troubles again. Techs continue to investigate.

    10:34 am: Service restored again. Techs continue to investigate root cause.

    10:24 am: Users are again reporting login troubles with OnBase Unity. Techs continue to investigate.

    8:37 am: Service restored, we continue to investigate the root cause.


    Wednesday, 2/17/16, 8:22 am

    AFFECTED SERVICES

    • OnBase Unity client

     

    IMPACT

    • Users reporting that they are unable to log in

Feb 17, 10:35 am: Tiffin Hall Internet, down for some users || RESOLVED

  • 10:56 am - Service restored after technician reset a malfunctioning piece of network equipment.


    AFFECTED SERVICES

    • Tiffin Hall Internet

     

    IMPACT

    • Down for some users
    • Technician currently on site to investigate

Feb 15, 10:02 am - 12:33 pm: TutorTRAC, down for all users || RESOLVED

  • Service restored after restarting a server.

     

    AFFECTED SERVICES

    • TutorTRAC
    IMPACT
    • Down for all users

Feb 14, 7:00 am - 7:30 am: Internet, degraded performance || RESOLVED

  • Outage caused by a DDoS attack and was resolved once our ISP filtered out the attack-related traffic.

     

    AFFECTED SERVICES

    • Internet
    IMPACT
    • Degraded performance

Feb 13, 7:36 am - 10:33 am: Many OHIO onilne services, down for some users || RESOLVED

  • Outages were unintended side effect of scheduled firewall maintenance. Most services were restored within 15 minutes, but some issues continued longer.


    AFFECTED SERVICES

    • A number of OHIO online services, including PeopleSoft, e-Biz, OBI, Pharos printing, VMware/VDI, and Blackboard Transact.

     

    IMPACT

    • Down for some users

Feb 11: 10:30 am: Fac/staff printing to Pharos || RESOLVED

  • Fac/staff unable to print to some Pharos-based printers. Restarting the printer seems to resolve the issue.

Feb 11, 8:34 am: Intermittent issues with multiple services || RESOLVED

  • 9:30 am: Service has been restored. Actual extent of outage was limited to AdAstra and Pharos mobile printing (myprincenter.ohio.edu). Both appear to have been side effects of yesterday's data center outage. Network slowness was caused by a denial of service attack against OARnet, our Internet service provider between 8-8:30 am.


    Thursday, 2/11/16, 8:34 am

    AFFECTED SERVICES:

    • Wired & wireless Internet

     

    IMPACT:

    • Intermittent interruptions
    • These interruptions are affecting end user connections as well as data center connections, so the problem can appear to be either a complete lack of Internet access or an inability to log into a central service like PeopleSoft SIS

Feb 10, 4:34 pm - 6:06 pm: Most OHIO online services || RESOLVED

  • Wednesday, 2/10/16, 4:44 pm - 6:06 pm
     
    AFFECTED SERVICES
     
    Most OHIO online services, including
    • Blackboard Learn
    • OHIO Front Door
    • PeopleSoft SIS & My OHIO Student Center
    • Oracle e-Business
    IMPACT
    • Down for all users
    ADDITIONAL DETAILS
    • Technicians are actively troubleshooting an issue with the university data center
    • Existing web sessions for cloud hosted services like Catmail, Qualtrics and Box appear to be unaffected; however, if you log out of the service, you may not be able to log back in.

Feb 5, 8:39 am - 8:47 am: Titanium || RESOLVED

  • Friday, 2/5/16, 8:39 am - 8:47 am

    Service restored after restarting file storage that did not properly come back online after an automated system update.

     

    AFFECTED SERVICES

    Titanium (Counseling Services app)

     

    IMPACT

    Down for all users

February 4, 5:36pm Catmail || RESOLVED

  • Thursday Februrary 4, 2016 6:00pm: Service restored.  Outage was caused by a lack of disk space on the shibboleth server.


     

    Thursday February 4, 2016 5:36pm

     

    AFFECTED SERVICES:

    Catmail

     

    IMPACT:

    Some users are reporting that they receive a redirect error when attempting to login.

Feb 2: 12:12 pm - 12:18 pm: Identity manager/iTIM || RESOLVED

  • Service restored at 12:18 pm. Techs have identified the function that caused this outage and have updated the system to prevent future outages.


    AFFECTED SERVICES

    • Identity Manager
    • Password changes
    • Account activations

     

    IMPACT

    Down for all users 

Jan 31, 2:35 pm: Dougan Hall wired & wireless Internet || RESOLVED

  • Service restored at 4:10 pm. Outage caused by a faulty hardware module. Module has been replaced.

     

    AFFECTED SERVICES

     

    Dougan Hall wired & wireless Internet

     

    IMPACT

     

    Down for all users

Jan 30: Multiple web apps on webapps.ohio.edu || RESTORED

  • AFFECTED SERVICES
     
    • Course Offerings
    • Faculty and Advising Center
    • OCEAN
    • CERT
    • Textbook
    • Authorized Access
    • Online Graduate Appointments
    • Bobcat Student Orientation
    • GRADS
    • Financial Aid Online Services
    • Scholarship Management System
    • CTL
    • Guest Network Registration
    • CEMS
    • Application for Graduation
    • Alumni

    IMPACT
     
    • Down for all users
    • Server did not come back up properly after an automated system update. Service restored after rebooting the affected server.

Jan 30: Netreg || RESTORED



  • AFFECTED SERVICES

     





     

    Network registration web tool


    IMPACT  


    Down for all users
     




    Server did not come back up properly after an automated system update. Service restored after rebooting the affected server.








     

     

Jan 30: Internet access, intermittent

  • 10:47 am - OARnet is applying technical measures to filter out the denial of service attack traffic

     

    AFFECTED SERVICES
     
    Internet access
    • Users on the university network unable to access sites outside of the university 
    • Off campus users unable to access university-hosted sites like Blackboard Learn, PeopleSoft SIS, Oracle e-Business, the OHIO Front Door website, etc.

    IMPACT
    • Intermittent interruptions
    • Technicians from OHIO's Internet provider (OARnet) currently are investigating multiple issues, including a possible denial-of-service attack that is consuming the university's available bandwidth
     

Jan 29: 8:47 am: Student printing at Dublin, Southern & Proctorville || RESOLVED

  • 1:34 pm - Proctorville restored. Root cause at all 3 locations: servers did not properly come back up after an automated system update. Physical reboot was required.

    9:22 am - Southern restored. Tech en route to Proctorville to restore service there.

    8:49 am: Dublin campus service restored. Other locations should be restored once we reboot their servers.


    AFFECTED SERVICES

    • Student printing (Pharos) at Dublin, Southern & Proctorville

     

    IMPACT

    • Service down for some users

Jan 29, 1:40 am - 2:15 am: Internet access || RESOLVED

  • Friday, Jan 29, 1:40 am - 2:15 am

    AFFECTED SERVICES

    • Internet
    • Access to off-campus sites from the campus network
    • Access to university-hosted services from off-campus (Blackboard Learn, Front Door web server, etc.)

     

    IMPACT

    • Service down for all users.
    • This appears to have been an issue with our Internet provider. OARnet techs are investigating.

     

Jan 28, 10:57 am: Konneker Research Center network || RESTORED

  • Service restored, 12:23 pm


    Thursday, 1/28/16, 10:57 am

    AFFECTED SERVICES:

    Konneker Research Center

    •  Wireless Internet (WiFi)
    •  Wired Ethernet
    •  Wired telephones

     

    IMPACT:

    Services are completely down due to a water leak affecting network equipment.

Jan 26, 10:30 am -10:40 am: Web pages on ww2.ohio.edu || RESTORED

  • Service was restored when the underlying web server was restarted. 

     

    AFFECTED SERVICES

     

    Web pages located on the static page server (ww2)

     

    IMPACT

     

    Service down for some users:

    "No data was returned" error for pages hosted on ww2.ohio.edu for approximately 10 minutes.

Jan 22, 2:30 pm: Campus telephone lines, intermittent connection issue || RESOLVED

  • 1/26/16, 10:20 am: We have been unable to reproduce the problem and have received no user reports since yesterday. Techs will continue to monitor the telephone system in case the problem reappears.


    1/25/16, 3:58: We continue to look into isolated, intermittent cases where callers have difficulty connecting with off campus numbers. Telephone service carriers have been engaged to help troubleshoot.

    AFFECTED SERVICES:

    • University telephones

     

    IMPACT:

    • Currenlty investigating reports of intermittent problems with 'dead air' on calls between university lines and off campus

Jan 22, 12:15 pm: IMAP access to Catmail, degraded performance

  • Friday, 1/22/16, 12:15 pm

    AFFECTED SERVICES

    IMAP access to Catmail

    IMPACT

    Degraded performance for some users

    Microsoft currently is working on fixing this issue. Status update from Microsoft:

    Incident EX41924*
    *Affected product:* Office 365 Exchange Online [Catmail]
    *Current state:* Restoring service

    *Current Status:* Engineers completed the development phase of the fix. Subsequently, engineers tested the fix and confirmed that it restores service. Engineers are now deploying the fix to the affected environment.

    *User Impact:* Affected users are unable to receive email messages when accessing Exchange Online through Internet Message Access Protocol (IMAP). As a workaround, users may be able to access email via Outlook on the web. [Catmail: https://catmail.ohio.edu ]

    *Customer Impact:* Analysis of incident scope indicates that a limited number of customers appear to be impacted by this event. However, those customers affected likely have a large number of users experiencing impact. Engineers have received a few isolated customer reports of this issue.

    *Incident Start Time:* Monday, January 18, 2016, at 12:00 AM UTC

    *Estimated Restoration Time:* By Saturday, January 23, 2016, at 2:00 AM UTC

    *Preliminary Root Cause:* As part of our efforts to improve service performance, an update was deployed to a subset of code which is responsible for obtaining the subscribed folder list. However, the update caused a code issue that prevented the list from being automatically loaded.

    *Next Update by:* Saturday, January 23, 2016, at 2:00 AM UTC

Jan 21, 2:24pm: Walter Fieldhouse WiFi, service degraded for some users || RESOLVED

  • 2:39 pm: Service restored after rebooting a network switch


    Thursday, Jan 21, 2:24 pm

    AFFECTED SERVICES

    Walter Fieldhouse WiFi

    IMAPCT

    Service degraded for some users

     

Jan 14, 7:30 am - 8:00 am: SIS apps including ALUMNI, Application for Graduation, CEMS, CTL, Guest Network Registration, GRADS, and transcripts || RESOLVED

  • Resolved at 8:00 am. Outage was caused by a lack of system resources on the host server. This only became an issue after deploying several new apps without adding resources. To prevent a recurrence, we will be upgrading the server on Jan 16.


    AFFECTED SERVICES:

    SIS apps including ALUMNI, Application for Graduation,  CEMS, CTL, Guest Network Registration, GRADS, and transcripts

     

    IMPACT:

    Services are completely down

Jan 13, 9:25 am: Show & Share video access for OULN

  • Wednesday, 1/13/16, 9:25 am

    AFFECTED SERVICES

    Show & Share video access for OULN students

    IMPACT

    Students unable to view videos on Show & Share.

    OULN staff are working on the issue.
     

Jan 12, 6:10 pm - 7:44 pm, Convocation Center WiFi || RESOLVED

  • Service resoloved after restarting a wireless controller. Tech noted that the temperature in the closet housing the controller was well below normal operating range for that kind of equipment.


    Tuesday, 1/12/16, 6:10 pm - 7:44 pm

    AFFECTED SERVICES:

    Convocation Center

     Wireless Internet (WiFi)

    IMPACT:

    Services are completely down

Jan 11, 8:49 am - 9:00 am: Network registration tool

  • Monday, 1/11/16, 8:49 am - 9:00 am - RESOLVED


    AFFECTED SERVICES:

     Online tool for registering devices on the wired network (NetReg)

    IMPACT:

    Service is completely down

January 6, 2:30pm-2:55pm Catmail OWA||RESOLVED

  • 3:44 pm - Microsoft has released the following Office 365 outage report. They are listing the problem as resolved, but no root cause.

    Incident: MO41617

    Final Status: The investigation determined that there was a temporary issue affecting routing of requests to the Microsoft Online Portal. This issue has been resolved and engineers have confirmed that the service is no longer impacted. The root cause is under investigation.

    User Impact: Affected users were unable to sign in to the Office 365 service.

    Customer Impact: Analysis of incident scope indicated that many customers appeared to be impacted by this event. For those customers affected, this issue potentially affected any user attempting to log in to the service. Some customers reported that their organization was affected by this issue.

    Incident Start Time: Wednesday, January 6, 2016, at 19:21 UTC

    Incident End Time: Wednesday, January 6, 2016, at 19:50 UTC

    Preliminary Root Cause: The root cause of this incident is not known but is under investigation. Based on the impact of this incident the root cause will be provided, if identified, within 30 days.

    Next Steps: The following is a list of known action item(s) associated with this incident. As part of the Office 365 problem management process, additional engineering actions may be identified to improve the overall service.

    Action: Review the monitoring infrastructure for improvements as this event was reported by customers before the investigation prompted a communication to be published to the Service Health Dashboard.

    Action: Identify the underlying root cause and develop a solution to prevent reoccurrences.



    Subject: [OIT Notice] Catmail OWA connectivity issues, January 6, 2015

    Wednesday, January 6, aprroximately 2:30pm to 2:55pm

    AFFECTED SERVICES:
    Office 365 services including:
    Catmail OWA (Web page version only)

    IMPACT:
    Services were down for users that go to catmail.ohio.edu on a web page to access their email.
    Those that use Outlook or other desktop application should not have been affected.

    --
    Regular updates will be posted to https://ohio.edu/oit/outage and https://twitter.com/OhioIT 

    If you have any questions, contact the OIT Service Desk at https://www.ohio.edu/oitech or 740-593-1222.

Jan 5, 1:00 pm - 2:01 pm: Business Systems Web Apps || RESOLVED

  • January 5th, 2016 1:00pm - 2:00 pm (Service restored at 2:00 pm)

    AFFECTED SERVICES:
    New logins to BobcaBUY
    New logins to Concur
    CC Lookup
    Employee Lookup
    Natural Account Lookup
    Mail Barcodes
    CHAOS
    EMS
    FA View
    Mass Element Entry
    PEDS
    PO Summary View
    FMS RDM
    Student Rapid Hire
    Course Offerings
    Faculty and Advising Center
    OCEAN
    CERT
    Textbook
    Authorized Access
    OGA
    Bobcat Student Orientation
    GRADS
    Financial Aid Online Services
    Scholarship Management System
    CTL
    Guest Network Registration
    CEMS
    Application for Graduation
    Alumni

    IMPACT:
    Service was completely down for one hour.
    Service restored at 2:00 pm; however, we still are investigating the root cause.

Jan 2, 7:01 am: Intermittent issues with some applications || RESOLVED

  • 10:25 am - Service restored. All outages were caused by a portion of our central storage system not properly coming back online after unrelated, scheduled maintenance earlier this morning. Restarting the affected system resolved the outage.


    10:03 am - NetReg application currently showing some errors.


    8:04 am: e-Biz and Front Door should be performing normally. Vendor engaged to troubleshoot issue with network storage device.


    We are experiencing some intermittent issues this morning with our central network storage system. Brief outages for the OHIO Front Door and e-Biz.

Jan 2, 6:00 am: Blackboard Learn || RESOLVED

  • Service restored at 1:39 pm.

    Technicians continue to investigate the cause of this outage.

    If you notice any issues, contact the OIT Service Desk at https://www.ohio.edu/oitech or 740-593-1222.
     


    Saturday, Jan 2, 6:00 am
    AFFECTED SERVICES:
    Blackboard Learn
    Panopto Lecture Capture
    IMPACT:
    Service is completely down

Dec 22, 2:24 pm: Funds Available View, degraded performance || RESOLVED

  • 3:11 pm: Service restored after rebooting the affected system. Root cause still being investigated.


    Tuesday, 12/22/15, 2:24 pm

    AFFECTED SERVICES:

    OU Funds Available

    IMPACT:
    Slow and/or degraded performance
    System currently unable to load documents via OnBase

Dec 21, 8:00 am: Web interface for submitting Facilities work orders || RESOLVED

  • Issue Resolved at approximately 2:35pm


    Issue Recurrence at approximately noon. Technicians are once again looking into the issue.

    Issue Resolved at approximately 11am.

    The web for that allows users to submit a Facilities work order through the Facilities web site currently is not working. This is a side effect of the weekend's CommonSpot upgrade. Technicians are working to restore the feature.

Dec 21, 8:47 am: Lancaster campus Herrold 215A & 271 network || Resolved

  • Issue Resolved at approximately 1:40pm

     

    Network down in Herrold 215A & 271. Tech has been dispatched to investigate an unresponsive piece of network hardware.

Dec 19, 9:54 pm: VDI Blast || RESOLVED

  • Issue resolved after server reboot. Security cert was recently replaced that requried a restart, but the automatic restart did not happen properly.


    VDI Blast is unavailable for some Engineering users. Server reboot scheduled for 12/20 at 12:01 am to address the issue. Less than 1 minute interruption for anyone in a VDI session when the reboot happens.

Dec 15, 1:00 pm: Footprints rejecting attachments || RESOLVED

  • Service restored at 1:57pm after increasing disk space allocation for the server.


    The IT work order tracking system, Fooprints, currently is rejecting attachments with an 'out of space' error.

Dec 15, 10:21 am: Network & VoIP, portions of 4th floor Alden Library || RESOLVED

  • Service restored at 10:39 am. Outage caused by an electrical power failure in a communications closet.


    Tuesday, Dec 15, 10:21 am

    AFFECTED SERVICES:
    Wired Internet, WiFi and VoIP telephones in parts of 4th floor Alden. Technician on the way to investigate.

    IMPACT:
    Service is down for at least some users

Dec 14, 6:40 pm: eBiz, BobcatBUY and Funds Available View || RESOLVED

  • Slowness appears to have been related to specific user queries that were running overly long. System now has a process to detect and deal with such queries.


    Monday, 12/14/2015, 6:30pm

    AFFECTED SERVICES:
    Ebiz
    BobcatBUY
    Funds Available View


    IMPACT:
    Slow and/or degraded performance

    Services will be down for a reboot at 7:30 pm (less than 15 minutes)

     

Dec 14, 9:10 pm - 9:30 pm: Reboot for eBiz, BobcatBUY, Concur, etc. || RESOLVED

  • Work completed. Original slowness appears to have been related to specific user queries that were running overly long. System now has a process to detect and deal with such queries.


    Monday, Dec 14, 2015, 9:10 pm

    20 minute outage to perform a full reboot of Oracle e-Business and related apps to address earlier reports of slowness

    AFFECTED SERVICES:
    Oracle E-Business Suite:
      Oracle HRMS
      Oracle FMS
      Employee Management System
      OUFMS Reporting
      My Personal Information
    BobcatBUY
    Online Graduate Appointments
    OU Business Online
    OU Funds Available
    OU PO Summary View
    Concur
    Student Rapid Hire

    IMPACT:
    20 minute outage for all listed services


Dec 11, 12:15 pm: OnBase Document Management System, Users of the Unity and Application Enabler Clients [RESOLVED]

  • Friday, 12/11/2015, 8:30 AM


    AFFECTED SERVICES:
    OnBase Unity Clients
    OnBase Application Enabler Clients


    IMPACT:
    Service is down for at least some users


    A reset is scheduled for 12:15 PM and should only take a minute to complete

Dec 10, 9:50 am - 5:15 pm: Funds Available View, degraded performance || RESOLVED

  • Service restored at 5:15 pm after restarting one of the servers that supports e-Biz.


    Thursday, 12/10/2015, 9:50 AM

    AFFECTED SERVICES:
    Funds Available View

    IMPACT:
    Slow and/or degraded performance

Dec 8, 8:10 am - 8:37 am: Astra room scheduling || RESOLVED

  • Astra room scheduling was down from 8:10 am - 8:37. Service was restored after the server was rebooted.

Dec 8, 10:37 am: Transferology Connector || RESTORED

  • 1:40 pm, Dec 9: Service restored.


    8:00 am, Dec 9: Non-students who need to access DARS continue to be unable to do so. Typical volume of such requests is around two per week.


    5:31 pm: Vendor continues to troubleshoot. The only portion of the service that is down is DARS requests from non-students. Majority of Admissions services are available.


    1:20 pm: Service remains down. Vendor support actively engaged.


    UNPLANNED OUTAGE:

    • Tuesday, Dec 8, 10:37 am

    AFFECTED SERVICES:

    • Transferology Connector (Recruiting)

    IMPACT:

    • Service is down for at least some users
    • Vendor has been notified and is troubleshooting

Dec 4, 3:30 pm: Intermittent Internet issues || RESTORED

  • Service restored at 3:42 pm. Caused by a malfunctioning DHCP script that prevented users from renewing their IP address leases or obtaining new ones.


    Technicians currently are troubleshooting an unplanned IT service outage.

    AFFECTED SERVICES:

    Wired & wireless Internet

    IMPACT:

    Service is down for at least some users

    ESTIMATED TIME TO RESTORE:

    unknown

Dec 4, 11:14 am: VPN access from virtual machines || RESTORED

  • 8:00 pm - Service restored.


    4:35 pm: Technicians continue to troubleshoot this issue.


    We are investigating possible issues with VPN Pulse connections for users who are logged into a VM session.

Dec 4, 12:34 pm: Seigfred first floor network outage [RESOLVED]

  • Network outage for the first floor of Seigfred.

    First report was around 12:34 pm.

    Service restored around 1:16 pm.

    Cause was a tripped elecrical circuit breaker.

Dec 2, 9:10 am: Some OBI reports not working || RESOLVED

  • Service restored as of 9:16 am after we corrected a configuration error.


    We are investigating reports of some OBI reports returning a database error.

    IMPACT:

    • Payroll-specific OBI reports are unable to run
    • e-Biz and PeopleSoft reports are not affected

     

    ESTIMATED TIME TO RESTORE:

    Unknown

Dec 2, 8:00 am: New student OHIO ID activations || RESOLVED

  • 8:33 am: Service restored after restarting a server that had a memory issue.


    The app that admitted students use to activate their OHIO IDs currently is down.

    IMPACT:

    Service is completely down

    ESTIMATED TIME TO RESTORE:

    Unknown

Dec 2, 12:40 pm: Edwards Accelerator Lab Network Issues || RESTORED

  • Services Restored at 1:20pm after a broken piece of network hardware was replaced.


    We are investigating reports of some network outages in the Edwards Accelerator Lab.

    IMPACT:

    • Intermittent wired and wireless connection issue in the building

     

    ESTIMATED TIME TO RESTORE:

    Approximately one hour

Dec 1, 4:10 pm: Intermittent network & VoIP performance, Dublin campus || RESTORED

  • 7:26 pm - Service restored after we disabled a malfunctioning network port.


    AFFECTED SERVICES:

    Dublin campus:

    • Wired Internet
    • Wireless Internet
    • VoIP phones

     

    IMPACT:

    • Slow and/or degraded performance
    • Intermittent, and only affecting a subset of users

     

    ESTIMATED TIME TO RESTORE:

    • Unknown
    • Possibility of brief outages to troubleshoot. If needed, these will be scheduled for outside normal class and office hours.

     

Nov 20, 6:45 am: Authentication problems with WiFi || RESOLVED

  • 2:37: Service fully restored:

    • OHIO University and OHIO University Guest have been up and working properly since 11:30 am
    • eduroam for OHIO users came back at 1:35 pm and for non-OHIO users at 2:37 pm
    • Root cause still being investigated

    12:45 pm - The OHIO Univeristy and OHIO University Guest wireless networks should be working properly. The "eduroam" wireless network remains offline.


    12:02 pm - Login server issues continue to cause intermittent connection problems for @ohiou WiFi. Working with vendor to resolve.


    10:55 am - Still troubleshooting. Vendor support now is involved. Users may continue to have intermittent problems connecting to the university's WiFi.


    9:51 am: Still working on the problem. Users may continue to have intermittent problems connecting to the university's WiFi.


    Technicians currently are troubleshooting an unplanned IT service outage.

    AFFECTED SERVICES:

    • Login/authentication servers

     

    IMPACT:

    • Users are intermittently unable to log in to the wireless network

     

    ESTIMATED TIME TO RESTORE:

    Unknown

Nov 18, 8 am: e-Biz and related apps || RESTORED

  • 11:18 am: Service restored following reboot; however, we have not yet identified what caused the outage. Ok to resume using the system, but save your work often.


    We will reboot e-Biz at 10:50 am. Service will be comeplely down for approximately 30 minutes.

    AFFECTED SERVICES

    Oracle E-Business Suite:

      Oracle HRMS

      Oracle FMS

      Employee Management System

      OUFMS Reporting

      My Personal Information

    BobcatBUY

    Online Graduate Appointments

    OU Business Online

    OU Funds Available

    OU PO Summary View

    OBI reports


    Technicians currently are troubleshooting an unplanned IT service outage.

     

    AFFECTED SERVICES:

     

    Oracle E-Business Suite and applications that query e-Biz, like OBI.

     

    IMPACT:

     

    New logins and query connections are not working.

     

    Issue began around 8 am on Nov 18.

     

    ESTIMATED TIME TO RESTORE:

     

    Unknown