ITS Help Desks

The circuit provider reports that they saw errors in the time frame but cannot determine the root cause.

The database upgrade is complete. Outage window spanned 5:07 to 6:02 a.m. All applications appear to be functional.

11/10/09 4:45pm

On 11/13/09 during the maintenance window TNS will be performing maintenance on the Computer2 router. Network connectivity may be interrupted briefly while the maintenance is performed.

The CIDR application will be restarted at noon hour today to correct a problem related to the 2nd factor authentication.

On Tuesday, November 24, during the regularly scheduled maintenance window (5:00-7:00 a.m.), Information Technology Services (ITS) will upgrade the sever software that supports the Jabber chat service, chat.psu.edu. During this time, intermittent outages may occur; however, total outage time is not expected to exceed fifteen minutes. As a result of the upgrade, Jabber will be ready to use/release as a full production service. Public announcements are scheduled to follow.

Information about Jabber and instructions for configuring chat clients to use chat.psu.edu are found via the "Penn State Jabber Client" section of the ITS Knowledge Base (KB) at .

Information and requests for assistance should be directed to ITS Help Desk Staff at helpdesk@psu.edu.

On Thursday, November 12th, 2009, TNS will upgrade the router at the Hershey Medical Center and College of Medicine during the maintenance window. All services will be restored by 7:00am.

As of 11:57, both email.psu.edu and Webmail are functioning as expected.

PAWS will be available the whole time

Earlier this morning, customers may have experienced delays in regards to DNS service. Upon investigation, we have identified and corrected the root cause.

Email notifications of Course Submission & Consultation System (CSCS) consultation requests are now being properly delivered. The specific problem was that email notifications were not being sent to the individuals asked to consult, but instead were sent to the individual creating the course proposal and erroneously labeled as test emails.

Information Technology Services (ITS) corrected the problem effective 6:00 a.m., November 4. ITS regrets any inconvenience this problem has caused.

For more information, please contact ANGEL Support.

Summary:
Complications during a planned hardware upgrade of a CIFS Gateway led
to an outage outside the established maintenance windows, and degraded
some web services.

Details:
At 7:00AM, Wednesday November 04, one of the six CIFS gateway servers
became stuck in a bad state while attempting a graceful shutdown.
Said state caused a steady increase in load on those PASS nodes
attempting to access the filesystems being touched by the bad CIFS
gateway. As a result, each of the CIFS gateways gracefully removed
themselves from the loadbalancer after crossing the maximum load
threshold, resulting in the overall service outage.

In addition, www.personal.psu.edu and php.scripts.psu.edu are known
to have been impacted, experiencing service degradation due to the
state of those filesystems being touched by the PASS node in a bad state.

All services were restored to full functionality at approximately
7:50AM. Debug data has been gathered and we are working with the vendor
to permanently resolve the root cause of this issue.

On Thursday, November 5th, 2009, TNS will upgrade the router at Erie - The Behrend College during the maintenance window. All services will be restored by 7:00am.

On Tuesday, November 3rd, 2009, during the maintenance window, TNS will replace a piece of hardware reporting errors in Telecom4. During this time, the following campuses will be on redundant connections back to University Park for approximately 5 minutes:

Hazleton
Scranton
Schuylkill
Wilkes-Barre
Hershey
Dubois
McKeesport
Freeport
Northpointe
Shenango
Beaver
New Kensington
Fayette

The GI services will be restarted between 3:00am and 4:00am Monday, 11/2/2009. This is to allow for an enhancement to the workflow system.

Services. There will be an intermittent interruption of services calling the GI during this period.

On 11/4/09, during the maintenance window, TNS will be upgrading hardware in the telecom9 router. Network connectivity to the Telecom9 router will be disrupted while the maintenance is in process. All services will be returned to normal before the end of the maintenance window.

On 11/3/09, during the maintenance window, TNS will be upgrading hardware in the Computer2 router. Network connectivity to the Computer2 router will be disrupted while the maintenance is in process. All services will be returned to normal before the end of the maintenance window.

The following library services will be unavailable during the 5-7 a.m. maintenance window on Tuesday, November 3 while DLT applies a patch.

ABLE - UL Budget Request
Ask! E-mail reference services
Administrative Action Plan
AgNIC Youth Development
ARL Stats
Digitized Collections at Penn State website
Portions of PA's Past: Digital Bookshelf at Penn State
Employee Directory website
Employee Directory Maintenance
HR Employee Directory Maintenance
Endowments Application
E-newspapers title search
Erlic
HR Part Time Application
Instructional Programs Classes
IT Workplan
LFO Ballots
LLS Room Finder
Microform Indexes
EMS: Abandoned Mines of PA
Oliver
Promotion and Tenure Recognition Site!
Signage Form
Selectors Listing
Serials Review
TMI database


Along with:
Helpdesk
Alumni Database Authentication
Course Description Archive Application
EPIC

Networking services to Penn State Hershey Medical Center and College of Medicine have resumed to normal operation as of 7:46 am.

By 9:30 p.m. the problem has been cleared, and all services should have been restored. It appears that www.personal.psu.edu and at least some of the PASS CIFS gateways were affected.

ITS staff have taken diagnostic information from the failing servers and relayed it to the vendor for analysis.

The e-mail server was responding as of 4:25. WebMail was also responding as of 4:25. We are still looking into the cause of the outage. All systems are functioning normally as of 4:25.