Cover Story

When do you Take the Plunge?

VCU bridges the gap between old and new with hybrid integrated communications infrastructure.

For enterprise CIOs and telecom directors, the question of when to consider a migration from traditional private branch exchange (PBX) or Centrex systems to IP-based telephony is all too familiar. For Virginia Commonwealth University (VCU), CIO Mark Willis knew the decision would impact more than 45,000 students, faculty and staff spread across 120 buildings on two campuses located in the heart of Richmond, Va. In addition to the Monroe Park campus, VCU's Medical College of Virginia campus includes the VCU School of Medicine and a 780-bed hospital that has its own IT infrastructure and organization.

"We were a Centrex shop for years and our Monroe Park campus is in the middle of a residential area," says Willis. "As we have grown, we've taken older neighborhood buildings, converted them to academic space and incrementally added Centrex lines for phones. We were spending a lot of money, and we had some service issues in terms of how quickly we could respond to the needs of users."

In managing the old Centrex system, the VCU telecom team was relegated to the role of middleman, translating and passing customer requests for new phones and services to their provider (Verizon) for implementation. "Some departments were managing their own key systems thinking they could save money and add features we couldn't deliver easily," says Willis. The resulting patchwork created issues with university-wide features such as integrated voice mail and five-digit dialing.

In addition to the need to reduce costs and improve service for customers, Willis' team also had a wish list of improvements to the network's functionality, security and enhanced 911 (E-911) readiness. One final pain point ultimately tipped the scales in favor of a major change: The manufacturer of the old key-system phones had gone out of business, so getting replacement parts had grown difficult.

Willis tasked Bill Jones, a veteran VCU network architecture and project manager, with leading the effort to draft requirements for a new system that the university would manage in-house. Job one was to agree upon a system model; should they go with a traditional PBX or an all-IP telephony environment? Would a hybrid approach best meet the needs of their diverse environment?

The team settled on an approach that called for a voice-over-IP (VoIP) solution for the university and a digital PBX solution for VCU Health Systems, primarily due to initial VoIP reliability concerns for the hospital that Willis admits did not materialize–thanks to quality-of-service (QoS) standards in the network. A short-term decision also was made to leave the residence halls on the existing Centrex system, while more closely examining options for the future that take into account students' mobile phones, the university's existing wired and wireless broadband infrastructure, emergency communications requirements and other factors.

With the platform decision in hand, Jones created a comprehensive RFP that called for:

• a unified management platform;
• a system that could meet the current needs of both campuses and scale to meet future needs for five to 10 years without a forklift upgrade;
• potential for advanced features such as unified messaging;
• redundancy and failover capabilities;
• comprehensive 911 protection and accurate E-911 location information;
• an integrator that could handle a large implementation in a short time-frame; and
• a cost-effective solution that would meet the budget needs of a public university.

The duration of the initiative was critical due to the fact that VCU's technology services department also acts as a service provider, operating the university's telecommunications system much like an independent enterprise. "We're a telco company for VCU," says Willis.

The team's financial modeling determined the switch needed to take place relatively quickly so the projected cost savings could be used to help pay for the new system–rather than spent on multiple vendors during a lengthy transition.

"Finding a partner with experience implementing a project of this size ($11 million, 10,000 phones) was key for us," says Bob Neale, director of computing and communications for VCU technology services.

VCU's requirements led to the selection of a partner team of IBM Global Technology Services (integrator), Avaya (systems) and RedSky Technologies (E-911 location information). The system of choice would be the Avaya Communication Manager, a hybrid IP PBX that would support the mix of Avaya IP phones, digital phones and legacy analog phones that would be in place at the completion of the migration.

The convergence of voice and data required VCU to upgrade its network to support the QoS requirements for VoIP. As the university's primary data network partner, Cisco implemented virtual networks using multiprotocol label switching (MPLS) to handle the many different types of traffic the university's network now required. VCU is one of the first large universities to deploy MPLS across its campus network, according to Neale.

The convergence of voice and data required VCU to upgrade its network to support the QoS requirements for VoIP.

"We also wanted to re-engineer our data network to support a better security model," says Neale, who credits the project with helping VCU implement changes required by a new Virginia regulation that sets new security standards for networks containing sensitive data.

Combining a data network upgrade with a major phone system upgrade created challenges for the implementation team. "We knew we'd have to replace about a third of our 1,700 switches, and they all would have to be touched for QoS and VLAN configuration," says Jones. "And we had to do this with a very tightly controlled schedule."

"If we had it to do over again, we'd have started with the data network upgrade sooner to make implementation easier," says Willis.

To keep the complex process moving, the team developed a 14-week implementation schedule for each of 42 distinct implementation groups. The first four weeks of the cycle were devoted to network remediation and changes. The subsequent 10 weeks were spent meeting with users, determining needs, ordering, configuring and installing phones and, finally, switching phone numbers from the old Centrex lines to the new system.

"You had 14 groups at the various stages of the process at any given time and only one of those actually involved putting phones on the desk and cutting them over," says Willis.

To keep the intricate schedule rolling, the project team reserved a conference room in VCU's technology building for the entire 18-month process and held weekly face-to-face meetings involving all the partners. "We had Thursday ‘go/no go' cutover meetings each week because we had to tell Verizon by 9 a.m. Friday morning not to move the phone numbers if we weren't ready," says Neale. "If we weren't ready and didn't make the call, people wouldn't have a working phone."

Out of the 42 implementation groups, only two or three were delayed according to Jones. Given Verizon's scheduling requirements, any delay would push back implementation five weeks. The delayed buildings were rescheduled into subsequent implementation groups so the overall project schedule was not impacted.

The massive project also created the opportunity for VCU to assume responsibility for providing E-911 protection for students, faculty, staff and visitors to its two campuses. Taking control of this critical public safety, security and liability issue was a priority for VCU that became even more important after the shootings at Virginia Tech and the passage of legislation in Virginia requiring all organizations operating a PBX or multiline telephone system to implement E-911. Thirteen other states have E-911 legislation on the books.

Before the migration, Verizon managed the E-911 location information for the university's Centrex lines and could only track the location of a 911 caller to a network interface device (NID) that could serve one or several buildings.

"The best case (with the old system) was a 911 call could be tracked to a single building," says Jones.

Such 911 calls from the university or VCU health systems were routed through Richmond's public safety answering point (PSAP), which would then pass requests for police assistance to VCU's campus police department.

For VCU, the new system would have to do better. "On our medical campus, we have a 12-story, 500,000-square-foot building," says Willis. "How would you find someone if you only knew they were somewhere in that building?"

RedSky's E911 Manager was chosen to integrate with the Avaya Communication Manager to automatically capture location changes for all types of phones and update the regional automatic location identification (ALI) databases that provide location information to emergency dispatchers. To make sure the location information was accurate, IBM helped the implementation team conduct a complete location audit for every new phone, identifying it down to the room and jack level. The process uncovered several street addresses that were not E-911 compliant, which could slow response in an emergency.

With VCU's new network configuration, the location of a 911 caller can now be tracked at least to the floor within a building, even more specifically in larger buildings.

VCU also is using RedSky's emergency on-site notification (EON) feature to speed emergency response to 911 callers. With EON, campus police dispatchers are notified the instant a 911 call is placed and provided with the location of the caller.

"We get 10 to 15 911 calls per day," says Rachel Ross, communications manager for VCU campus police, one of the largest campus police forces in the country. "EON saves two to three minutes per call and helps us send help sooner."

One of the key benefits of VCU's new IP communications system is the ease with which moves, adds and changes (MAC) can be made. With campus expansions and renovations, Jones estimates that 25 percent of their phones are involved annually. Turnaround time for VoIP phone MAC activities has been reduced from two weeks to one business day for most tasks, thanks to centralized, in-house management.

CHANGES, UPDATES AUTOMATIC

Maintaining accurate E-911 location information in this constantly changing system would be an administrative nightmare if handled manually. The RedSky solution captures network changes the moment they occur and sends updates to the regional ALI database. "We don't have to worry about when a phone is moved to another place, it's automatic," says Jones.

For users, the project was virtually transparent until a new phone showed up on their desk. Behind the scenes, VCU technology services has had to learn a new way of doing business. The converged network now requires data and voice teams to work together more closely. One help desk number now handles all IT-related customer issues.

"Our data people were used to focusing on hardware, while our tele-group was used to talking with users all the time," says Neale. "We've had to educate one group on understanding users better, while the other group has had to come up to speed on technology. It's really been a challenge, but the staff has been up to the task."

The new system also transferred all maintenance responsibility to the technology services team. No longer can the team pass voice-related issues along to Verizon. Helping to make the transition smoother has been new centralized monitoring features that allow issues to be discovered before a customer calls.

With the main implementation phase recently concluded, Neale says the technology services team is now turning its attention toward leveraging its new system to improve business processes at the university. At an infrastructure level, these improvements will likely include the use of fax servers and expanded use of automatic call distributor applications. For individual users, the team plans to use marketing and training to make users more aware of the benefits of system features such as "find me" to route calls to users with multiple phones, and Web-based voice mail.

In addition, the new system is paying off financially. Jones estimates VCU has seen a 50 percent reduction in carrier costs by eliminating the majority of its leased lines. The university is also no longer paying one-time deployment charges for putting a phone on the desk, including handset costs, line fees, fees to set up voice mail and maintenance charges. Eliminating these charges and rolling them into the monthly per-line fee has simplified the billing process and allowed departments to establish more predictable budgets.

"Don't attempt to do something this massive without an outside partner," Neale advises. "We needed an outside partner to assist us, not only with the technology but to coordinate with all of the outside vendors necessary for the project. Some organizations might try to take on all of these activities themselves, but it would be nearly impossible to do. The co-existence and colocation of the vendor project team with our internal project team was expensive but worth it."

About RedSky Technologies

Tony Maier left AT&T Computer Systems in 1996 to launch RedSky Technologies, whose first applications worked with the PBX to capture the location of phones on the network and deliver that information to the regional databases that provided enhanced 911 (E-911) information to emergency dispatchers. Since that time, 14 states have adopted E-911 regulations and RedSky's E911 Manager has been installed in more than 300 enterprises, including 50 Fortune 500 companies, large banks, insurance companies, universities and government agencies.

RedSky also has launched a host of related products, including E911 Anywhere, a monthly service that tracks phones inside and outside the enterprise and routes 911 calls from anywhere in the United States and Canada.

For more information on RedSky Technologies (click here)