Keeping the U.S. Navy’s ships connected to each other, as well as to land-based Naval operations, can be a daunting task. The ships are usually on the move, necessitating satellite communications for these “road warriors” while away from their bases. With heavy traffic loads, high-speed connections are needed while at sea and at port. With such a network, unusual or unexplained network behavior can often be the norm. The U.S. Navy’s Pacific Region Network Operations Center (PRNOC) in Wahiawa, Hawaii, is in charge of maintaining the network infrastructure for ships based in the Pacific Ocean. The PRNOC provides ships with communication services both for military uses and for everyday educational, recreational and morale-related activities (e.g., Web surfing, online classes, e-mailing family and friends). While at sea, the ships’ onboard IP networks connect to the PRNOC via satellite links. When they come into port, they use fiber-optic “umbilical cords” to physically establish adjacencies.

Three other NOCs, also under the auspices of the Naval Network and Space Operations Command, provide similar services to fleets based in the Atlantic, Mediterranean and Indian Ocean regions. Each NOC houses the network hardware–routers, servers, firewalls, ATM or Gigabit Ethernet LAN devices–that provide the foundation for these services.

The PRNOC staff found that the Pacific Region network, which uses the open shortest path first (OSPF) routing protocol, would sometimes exhibit unusual or unexplained behavior. “When a ship would move from port out to sea and back, or from one ocean region to another, we would sometimes get odd network routes,” says Donn Murakami, a PRNOC fleet support engineer (FSE). “We’d hear complaints that a certain ship couldn’t reach some Web sites or other network services. Sometimes, a ship’s network couldn’t establish an adjacency to the network at all.”

To locate the problem, Murakami and the other FSEs would have to painstakingly log into router after router, checking the OSPF routing tables for misconfigurations or other problems. Dealing with widely dispersed geographic regions, different time zones, a large OSPF backbone network and multiple OSPF domains with numerous routers per domain made this a difficult and time-consuming task.

Ultimately, the FSEs might find that a ship had made an improper route announcement, inadvertently setting itself up as a default route through which all network traffic from other ships would pass (instead of each ship connecting directly to the PRNOC, as specified by standard operating procedures). Since that ship might have a slow satellite connection, bandwidth would quickly get clogged, leading to performance problems.

“Our customers–ships, submarines, mobile commands–would connect and disconnect at will, with an interior gateway protocol such as OSPF assuming that everyone is just one big happy family,” recalls Stanford Wong, also a PRNOC FSE. “OSPF works by flooding all connection information, in the form of OSPF link-state advertisements (LSAs), throughout the routing domain.

“Once in a while, a ship might inadvertently hook up to two different domains, which could overtax the routers. If we had high-speed connections to every ship, this wouldn’t be a huge problem. But with these satellite links, being flooded with OSPF LSAs clogs the pipes quickly, diminishing already limited bandwidth availability.”

Adding to the problem, Wong says, is that many of the installed routers are older versions with relatively low memory capacity. Upgrading or replacing the routers was not a viable short-term solution.

What the PRNOC staff needed was a clear, graphical picture of the OSPF network topology–a map of the network routes that would show them instantly where erroneous routes were causing problems.

“We wanted a box that would act like a router, participating in the OSPF LSAs and mapping the topology based on the OSPF announcements,” Wong says. “Then we could spot the culprit who was misconfigured, or see a failed attempt to establish an adjacency. We already had management tools that would tell us something had gone wrong. We needed something that would tell us where and how it had gone wrong, so we could go right to the problem and fix it.”

Wong conducted an Internet search to determine whether a commercial solution existed that would address his needs, turning up a product called Route Explorer, from Packet Design. Route Explorer lets users visualize, monitor and analyze an IP network’s logical (routing) operation, revealing otherwise hidden aspects of how OSPF and other routing protocols operate. It works by discovering all routers on the network, creating an end-to-end view of the routing topology, and “listening” to the protocol exchanges to learn when routing changes occur.

“We needed something that would tell us where and how it had gone wrong, so we could go right to the problem and fix it.”

Route Explorer was the only route-analytics offering, Wong found, that supported all four commonly used routing protocols–OSPF, border gateway protocol (BGP), intermediate system to intermediate system protocol, and enhanced interior gateway routing protocol. This could be an important consideration if the Navy ever considered using more than just OSPF in its network–for example, if it wanted to establish network peering via BGP with its Internet service provider, the Defense Information Systems Agency.

“We had Route Explorer up and running quickly, and saw that it mapped every OSPF link-state advertisement,” Wong says. “Then, it was just a matter of waiting for something to go wrong. When a ship changed its connection, we could immediately see on the screen the ‘ghost’ image of one connection fading away and the new one appearing. Once we knew which ship was causing this, we could e-mail them to correct the connection, or proactively cut off the connection from the NOC.”

Murakami adds, “As we watched, we saw ships doing things we weren’t anticipating, such as being on a physical and a satellite connection at the same time. Or a ship that had pulled into shore would still be on the satellite connection rather than the higher-bandwidth, fiber-optic one. The identification of a network issue that is impacting, or could potentially impact, a mission requires a timely and accurate report to personnel in the military chain of command.”

This capability can be invaluable when ships are depending on time-critical tactical services or when “military chat” is ongoing among a multinational coalition in a wartime action, or a situation such as the Asian tsunami relief effort, when critical video information was being transmitted around the clock. “The network services we provide to the fleet do have an impact on how fast they can respond to, analyze and resolve any situation,” Murakami says.

The PRNOC FSE staff, however, does not need to be sitting at the Route Explorer console waiting for problems to occur. The analyzer’s data store logs all routing events, including every OSPF link-state advertisement and routing anomaly. At any time after the fact, the FSE can play back the recorded events on the unit’s History Navigator, allowing troubleshooting of even those problems that are not currently occurring. Route Explorer also provides a forensic audit trail showing which error prevented an adjacency from being established and caused the route to repeatedly “flap” up and down.

This article was provided by Packet Design, Palo Alto, Calif.

For more information:
www.rsleads.com/601cn-259