ITS Help Desks

As of 11:57, both email.psu.edu and Webmail are functioning as expected.

PAWS will be available the whole time

Earlier this morning, customers may have experienced delays in regards to DNS service. Upon investigation, we have identified and corrected the root cause.

Email notifications of Course Submission & Consultation System (CSCS) consultation requests are now being properly delivered. The specific problem was that email notifications were not being sent to the individuals asked to consult, but instead were sent to the individual creating the course proposal and erroneously labeled as test emails.

Information Technology Services (ITS) corrected the problem effective 6:00 a.m., November 4. ITS regrets any inconvenience this problem has caused.

For more information, please contact ANGEL Support.

Summary:
Complications during a planned hardware upgrade of a CIFS Gateway led
to an outage outside the established maintenance windows, and degraded
some web services.

Details:
At 7:00AM, Wednesday November 04, one of the six CIFS gateway servers
became stuck in a bad state while attempting a graceful shutdown.
Said state caused a steady increase in load on those PASS nodes
attempting to access the filesystems being touched by the bad CIFS
gateway. As a result, each of the CIFS gateways gracefully removed
themselves from the loadbalancer after crossing the maximum load
threshold, resulting in the overall service outage.

In addition, www.personal.psu.edu and php.scripts.psu.edu are known
to have been impacted, experiencing service degradation due to the
state of those filesystems being touched by the PASS node in a bad state.

All services were restored to full functionality at approximately
7:50AM. Debug data has been gathered and we are working with the vendor
to permanently resolve the root cause of this issue.

On Thursday, November 5th, 2009, TNS will upgrade the router at Erie - The Behrend College during the maintenance window. All services will be restored by 7:00am.

On Tuesday, November 3rd, 2009, during the maintenance window, TNS will replace a piece of hardware reporting errors in Telecom4. During this time, the following campuses will be on redundant connections back to University Park for approximately 5 minutes:

Hazleton
Scranton
Schuylkill
Wilkes-Barre
Hershey
Dubois
McKeesport
Freeport
Northpointe
Shenango
Beaver
New Kensington
Fayette

The GI services will be restarted between 3:00am and 4:00am Monday, 11/2/2009. This is to allow for an enhancement to the workflow system.

Services. There will be an intermittent interruption of services calling the GI during this period.

On 11/4/09, during the maintenance window, TNS will be upgrading hardware in the telecom9 router. Network connectivity to the Telecom9 router will be disrupted while the maintenance is in process. All services will be returned to normal before the end of the maintenance window.

On 11/3/09, during the maintenance window, TNS will be upgrading hardware in the Computer2 router. Network connectivity to the Computer2 router will be disrupted while the maintenance is in process. All services will be returned to normal before the end of the maintenance window.

The following library services will be unavailable during the 5-7 a.m. maintenance window on Tuesday, November 3 while DLT applies a patch.

ABLE - UL Budget Request
Ask! E-mail reference services
Administrative Action Plan
AgNIC Youth Development
ARL Stats
Digitized Collections at Penn State website
Portions of PA's Past: Digital Bookshelf at Penn State
Employee Directory website
Employee Directory Maintenance
HR Employee Directory Maintenance
Endowments Application
E-newspapers title search
Erlic
HR Part Time Application
Instructional Programs Classes
IT Workplan
LFO Ballots
LLS Room Finder
Microform Indexes
EMS: Abandoned Mines of PA
Oliver
Promotion and Tenure Recognition Site!
Signage Form
Selectors Listing
Serials Review
TMI database


Along with:
Helpdesk
Alumni Database Authentication
Course Description Archive Application
EPIC

Networking services to Penn State Hershey Medical Center and College of Medicine have resumed to normal operation as of 7:46 am.

By 9:30 p.m. the problem has been cleared, and all services should have been restored. It appears that www.personal.psu.edu and at least some of the PASS CIFS gateways were affected.

ITS staff have taken diagnostic information from the failing servers and relayed it to the vendor for analysis.

The e-mail server was responding as of 4:25. WebMail was also responding as of 4:25. We are still looking into the cause of the outage. All systems are functioning normally as of 4:25.

All library systems have been restored.

At 10:06 a.m., both backend servers had reached maximum load. By 10:33 a.m., some of the gateways had been forced to use failover servers, and the load on all servers has returned to normal. User experience should have improved.

All communications problems have been resolved.
All AIS systems are now available.

On Sunday, November 15, 2009, from 5:00 until 11:00 a.m., Information Technology Services (ITS) will upgrade the Lightweight Directory Access Protocol (LDAP) server, which supports Penn State Directory Services, to Tivoli Directory Server version 6.2. The upgrade is being performed to enhance the performance and stability of the directory server. During this timeframe, several of the directory server's update-oriented services will be disabled; however, users will still be able to perform directory lookups via http://www.psu.edu/directory/. In addition, other directory-dependent services such as e-mail services, Penn State Access Account Storage Space (PASS), calendaring, and a variety of University-wide Web services will not be affected by this outage.

The directory-related services that will be disabled during the upgrade timeframe include:
*Directory entry and e-mail forwarding updates via the ITS Secure Server, https://www.work.psu.edu/
*Group creation via the User Managed Groups (UMGs) interface, http://umg.its.psu.edu/
*The Automatic E-mail Reply Message Utility (the "vacation" service) via https://www.work.psu.edu/vacation/
*The Student Organizations Server Administrative interface, https://www.clubs.psu.edu/

Information and requests for assistance may be directed to ITS Help Desk staff at helpdesk@psu.edu.

On Wednesday, November 11, 2009, during the regularly scheduled maintenance window (5:00-7:00 a.m.), Information Technology Services (ITS) will perform a hardware upgrade to the MySQL cluster systems that supports several University-wide services. Though this maintenance is being performed during the established maintenance window, several widely-used services may experience intermittent outages during this time. Services include:
*The Penn State Book Exchange, https://bookexchange.psu.edu/
*ITS News (administrative and public interfaces), https://news.its.psu.edu/admin/ and http://news.its.psu.edu/
*ITS Alerts (administrative and public interfaces), https://alerts.its.psu.edu/admin/ and http://alerts.its.psu.edu/
*ITS Downloads, https://downloads.its.psu.edu/
*The Blogs @ Penn State, https://blogs.psu.edu/ (Moveable Type system that supports The Blogs @ Penn State service)
*Chat services via Jabber (beta), chat.psu.edu
*Report Hate, https://reporthate.equity.psu.edu/

It is very likely that any outage time for services of this type will be very brief, perhaps only a few minutes, and the upgrade itself is expected to only take a few minutes. However, additional time has been allotted for any maintenance-related troubleshooting that may need to occur, and because the services noted above are widely used.

Inquiries and request for assistance should be directed to ITS Help Desk staff at helpdesk@psu.edu.

From 6:45 until 7:55 nfs.pass.psu.edu was not responding for NFS Version 3 mounts. The cause of the issue, was a restart of one of the supporting services to the NFS server. During this time, NFS version 4 was functioning properly.