Motivation

K-State's current campus network consists of a collapsed Ethernet backbone of routed subnets connecting buildings at 10 and 100 Mbps (see Figure 1). Core routers, consisting of Cisco AGS+, 2514, 7000, and 7513 models, are connected to each other at 10 and 100 Mbps with a Cisco 2900 switch. Built in the early 1990s, this design has served the campus well and has allowed for incremental improvements to alleviate localized bottlenecks. However, with the introduction of mediated instruction, the need for distributing video on and off campus, strategic use of the World Wide Web, participation in Internet2, a growing number of computational scientists, and the approaching year 2000, K-State is in serious need of replacing its outdated campus network

Summary of Proposal

• Establish a switched Gigabit Ethernet backbone network for the campus to replace the current 10/100 Mbps routed subnets. Backbone bandwidth increased 20 times (100 Mbps to 2000 Mbps).
• Connect three new core Cisco switches with Gigabit EtherChannel technology to get an effective backbone bandwidth of full duplex 2 gigabits per second (Gbps).
• Besides providing greater bandwidth, dramatic performance improvements can be achieved through switching packets at layer 2 rather than routing every packet at layer 3 of the network model. Cisco's "Netflow" technology will be enabled to help accomplish this.
• All but one non-y2k-compliant devices will be remove from the core. The Cisco AGS+ in West Hall will be the only remaining non-compliant network device and must be replaced in the next year.
• Several bottlenecks will be eliminated in the core (2 AGS+ routers and a backbone 10/100 switch)
• The network with the Human Resources servers will be upgraded from shared 10 Mbps Ethernet to switched 100 Mbps Fast Ethernet and connected directly into a core switch for significant performance improvement, especially during backups.
• Likewise, the main enterprise servers for the campus (e-mail, WWW, IBM S/390, DNS, UNIX, library) will be connected directly into a core switch at 100 Mbps. Spare ports are available to connect any of these servers with gigabit Ethernet.
• The Cisco 3000 switches serving Hale Library will be connected directly to a core switch, eliminating an extra network device in the path (a Cisco Catalyst 5000 that will be re-used to connect another building at 100 Mbps)
• Connect 21 more buildings at 100 or 1000 Mbps, adding to the 9 currently connected at 100 Mbps. This triples the number of buildings connected to the core network at high speed
• The Veterinary Medicine complex and other north campus buildings will have experience a dramatic improvement in the performance of their connection to the core campus network by replacing the old Cisco AGS+ router with a Cisco 7000 router and connecting it to the core with full duplex 200 Mbps Ethernet using Cisco's Fast EtherChannel technology. The three Veterinary Medicine buildings will connect to the 7000 at 100 Mbps as well.
• A limited level of fault tolerance will be available in the core with link redundancy (using the EtherChannel technology to have multiple links with fail-over between core switches and routers) and module redundancy (dual power supplies, switch processors, and route switch modules).
• Sophisticated network monitoring of the core network traffic will be available with an integrated RMON probe in the Power Plant 5500.

Requirements


        A new core campus network for K-State must address the following issues:

• Redundancy - Data should be automatically re-routed through an alternate path if the primary path is unavailable. Network devices should also have fault tolerance built into them in the form of redundant links, power supplies, processors, and other modules.
• Reliability - As the campus increasingly relies on the network to fulfill its mission and as that mission expands to cover any-time any-place learning, outages will be less tolerated. The network has to be available 24 hours a day seven days a week. Many factors play into providing a reliable network: the redundancy mentioned above, a reputable vendor, good technical support from the vendor, properly trained CNS staff, and the use of proven stable technologies
• Performance Scalability - New and more frequent uses of the campus network are pushing the need for increased performance. The core network must be able to grow accordingly.
• Flexibility - Campus networks are commonly a mixed set of technologies: 10/100/1000Mbps Ethernet, OC-3/OC-12/OC-48 ATM, hubs, switches, routers, and multiple vendors. A flexible core network design should support all current standard technologies and even anticipate future technologies.
• Quality of Service (QoS) guarantees - With the introduction of new applications requiring high bandwidth, like those identified in the NSF HPC grant for connection to the vBNS and/or Abilene networks, it is no longer possible to simply throw more bandwidth at the problem. The demands have outpaced our ability to "fatten the pipe," either technically or financially or both. This begs for managing the bandwidth instead of just increasing it. QoS guarantees can take the form of reservations (e.g., reserve 8 Mbps over the WAN link to the University of Nebraska at Lincoln between 10:30AM and 11:20AM for a live two-way video delivery of a course shared between the two institutions), prioritization (a real-time visualization may get preference over a file transfer), or guaranteed minimum latency (the network will not excessively delay delivery of your data).
• Multicasting - This is a very important technology to scale the distribution of high-bandwidth applications like video to a wider audience. Multicasting allows multiple sessions of a video stream (i.e., several students at different locations on and off campus) to share a single channel in the core network back to the server rather than each session having its own duplicate copy of the stream.
• Video distribution - Support is necessary for live video (one-way broadcasts of seminars, lectures, special events, two-way video conferencing, n-way video conferencing), video on demand, and varying quality up to full motion high resolution studio quality. This will require high bandwidth, multicasting, and QoS guarantees.
• Incremental implementation - Estimates of the cost of a complete overhaul of the campus network range from a minimum of $15 million to a maximum of over $30 million (the latter if all buildings are completely re-wired at a cost of about $500,000 each). Clearly, we will not have the money to do this as a single project, so it will have to be phased in over time. The solution should consist of focused incremental enhancements that make the most technical and fiscal sense at the time.
• Use existing cable plant - Due to time and money constraints, a new campus network must use existing single-mode and multimode fiber cables for the core and connecting the buildings. Obviously, many cables within buildings will need to be replaced to carry higher speeds, but those can be replaced as needed and therefore phased in over time.
• Support of existing Ethernet networks - Since the project will be phased in, existing shared 10 Mbps, switched 10 Mbps, switched 100 Mbps, and OC-3 ATM (connection to KANREN and Great Plains Network) networks must be supported. Furthermore, switched, and even in some cases shared, 10 Mbps networks will provide adequate bandwidth to many of the desktops on this campus for a long time to come. There is no compelling need to upgrade those connections at this time so they must be supported by the new backbone technology.
• Support for IP, IPX, and Appletalk protocols - While the great majority of the network traffic on the K-State campus is IP, we will still have IPX-based NetWare networks and Appletalk-based Macintosh networks for many years.
• Integrated services - Carrying data, video, and voice traffic over the same physical network can save money and possibly be easier to manage than physically separate data, video, and voice systems. This is especially true for WAN connections like the Salina campus, Konza Prairie headquarters, off-campus organized living units, and possibly even the University of Kansas.
• Security features - Human Resources servers are currently protected by a firewall and some filtering is done on data entering and leaving the campus through the Internet connection. Obviously these services must be retained and even expanded.
• Support K-State's commitment to Internet2 - This is primarily in the form of high bandwidth delivered to Internet2 researchers, and tracking and deploying newly developed technologies such as multicasting and QoS. The solution has to scale up to meet the growing demands of I2.
• Year 2000 compliance - Not only will the new devices and software have to be year 2000 compliant, but the current Cisco AGS+ routers must be taken out of operation by January 1, 2000, since Cisco has end-of-life'd that model and will not provide any y2k-compliant updates.

CNS proposes to replace the current campus network with a switched Gigabit Ethernet network consisting of up to six Cisco Catalyst 5500 series and Catalyst 4000 switches at the core (see Figure 2). In the initial phase, we propose to install a 5500 switch at the Power Plant, a 5509 switch in Hale Library machine room, and a model 4000 switch at College Court. The College Court 4000 and Hale 5509 will both connect to the Power Plant 5500 using Cisco's Gigabit EtherChannel technology to aggregate two Gigabit Ethernet links and get an effective bandwidth of full duplex 2000 Mbps (2 Gbps) between each switch. The Power Plant switch will replace the set of routers and switches that currently form the backbone of the K-State network. The Hale Library switch will connect the campus enterprise servers like the IBM S/390, E-mail, WWW, KATS, and UNIX. The College Court switch will connect the servers managed by the Division of Continuing Education (DCE) that are experiencing increasingly heavy use by distance students accessing web-based course materials and on-campus students and instructors accessing K-State OnLine (http://online.ksu.edu).

Later phases will add three more Gigabit Ethernet switches (or newer compatible technology) at the core. One will replace the router at the Veterinary Medicine complex that serves the buildings on the north end of the campus. Another will replace the Cisco AGS+ router in West Hall that serves the northeast part of campus and is need of replacement since it is not year2000 compliant. The third will be installed at the Foundation Center to connect servers at that location hosted by CNS and OIS. This will complete the Gigabit Ethernet backbone..

The Cisco Catalyst 5500 series switch is the next generation of the Catalyst 5000 series, with modules for 10/100/1000 Mbps Ethernet. Thus, existing 10 and 100 Mbps subnets can be connected to the new core without changing anything within the buildings, or be upgraded to Gigabit Ethernet where needed. The 5500 architecture scales to 50 Gbps with a throughput of millions of packets per second (pps).

This solution addresses the project requirements in the following manner:

• Redundancy - Link redundancy is achieved by aggregating multiple Fast Ethernets or Gigabit Ethernets using Cisco's EtherChannel technology. If, for example, you aggregate two Gigabit Ethernets and one fails, all traffic is switched through the single channel that did not fail. However, you cannot span EtherChannel across multiple modules (cards plugged into the chassis) - all interfaces participating in the aggregated EtherChannel must be on the same module. If an Ethernet module in a 5500 fails, all buildings connected to that module will be off the network. We could stock spares of each type of Ethernet module to minimize the downtime in the event of a failure. Device redundancy is achieved with redundant power supplies, switch processors, and route switch modules.
• Reliability - The Cisco 5500 switch is a mature product that evolved from an earlier highly successful Catalyst 5000 switch. The 4000 also evolved from the 5000 series, but is new technology. Likewise, Cisco as a company has been highly successful. K-State has a long relationship with Cisco, familiarity with its products and technical support, and has worked closely with their engineers on this and a number of earlier projects.
• Performance Scalability - The 5500 and 4000 switches support Cisco's proprietary EtherChannel that aggregates pairs of full-duplex Fast Ethernets or Gigabit Ethernets to scale from 200 Mbps up to 800 Mbps channels or 2 Gbps to 8 Gbps to other Cisco devices that support it (like the Cisco 7000 and 7513 routers and Catalyst 5000 and 5505 switches). EtherChannel also does load balancing over the aggregated Fast or Gigabit Ethernets. Finally, Cisco's "Netflow Switching" technology will allow a stream of packets such as a videoconference session to be routed at first (layer 3 service) and then switched at layer 2 once the flow is identified. This will dramatically improve performance for sustained sessions. In the proposed design, 2 Fast Ethernets will be aggregated to connect the Power Plant 5500 to the 7513 router (Internet connection), Foundation Center 5505 switch (CNS, ISO, Foundation, Alumni), and VetMed complex 7000 router. The three new Gigabit-capable switches will be connected to each other with two Gigabit Ethernet channels to yield full-duplex 2 Gbps channels.
• Flexibility - The 5500s and 4000 support full duplex switched 10/100 Mbps Ethernet with fiber or copper interfaces. Gigabit Ethernet (1000 Mbps) is supported with both Multimode and Single-mode fiber interfaces. Furthermore, Cisco as a company has a long history of involvement in the leading edge of developing new technologies. We therefore anticipate support for emerging technologies, but of course cannot rely on this.
• Quality of Service (QoS) guarantees - IP over Ethernet has no native QoS features like ATM does. However, much work is being done with QoS implementations over IP, such as RSVP and "differentiated services" ("DiffServ"). Cisco is involved in these efforts, including work on mapping RSVP to Catalyst 5500 priority queues. Internet2 QoS efforts are focused on DiffServ at this time and Cisco is involved in this work. For now, a switched gigabit Ethernet core will provide enough bandwidth to make QoS less pressing of an issue.
• Multicasting - Cisco developed the multicast standard and all of their devices support it.
• Video distribution - This solution relies on delivering video over IP, which is an active area of research and product development. Cisco is promoting the use of IP/TV with multicasting to distribute video on a campus over IP. An NT server with a large amount of disk space could be located by one of the three 5500s to function as an IP/TV server for the campus. To distribute the load on the core switches, it may be best to locate the IP/TV server in College Court.
• Incremental implementation - Three, or even just two or one, high speed switches can be installed now and all existing networks connected to the new core. Individual switches can be added to the core later as the need arises and funds are available to meet the goal of six core switches. Also, increasing bandwidth to buildings and/or desktops and servers can be added incrementally. In this design, spare Fast Ethernet and Gigabit Ethernet ports are available on the new core switches.
• Use existing cable plant - No new fiber cable will be needed in the core. Gigabit connections between the three core switches will require single-mode fiber since the distance limit for Gigabit Ethernet over multimode fiber is only 550 meters (limit for single-mode fiber is 2 miles to 5 km). Telecommunications said there is adequate fiber available.
• Support of existing network technologies - All Ethernet and ATM technologies currently used at K-State can connect to the new Gigabit Ethernet core.
• Support for IP, IPX, and Appletalk protocols - The Route Switch Module added to the two 5500 switches will handle all three protocols.
• Integrated services - Integration of multiple services is an active area of development, is strongly promoted by Cisco with their "Multiservice Access" line of products. However, we will have to rely on integration of the services over IP (voice over IP, video over IP) coupled with future developments in QoS over IP.
• Security features - The Cisco 5500 with the Route Switch Module fully supports the port-based access lists used for filtering the Internet connection on the existing 7513 router and the HR servers behind the 2514 firewall. In fact, we will be able to eliminate the 2514 and increase the bandwidth to the HR servers to 100 Mbps, something we have needed to do for a long time but could not because of the limitations of the 2514.
• Support K-State's commitment to Internet2 - Cisco is an original corporate sponsor of the Internet2 Project of UCAID and has been actively involved in I2 research projects. Cisco is also one of the corporate sponsors of the new Abilene Network. We can be confident that any I2 developments will be supported by Cisco hardware and software.
• Year 2000 compliance - All new Cisco hardware and software is y2k compliant. The first phase also eliminates two Cisco AGS+ routers (one in the power plant and one at the VetMed complex). The AGS+ at West Hall will have to be replaced within the next year.

Gigabit Ethernet vs. ATM

Most universities that upgraded their campus backbone network in the last few years implemented an ATM core network. The increased bandwidth and reliability over Ethernet-based networks was attractive and the promise of integrated services (voice, video, and data sharing the same lines) and Quality of Service led people to make the switch from frames to cells (Ethernet to ATM) in their backbone networks. While integrated services have been deployed extensively over WAN links, they have not seen wide acceptance in campus networks. Nor has the native QoS of ATM garnered widespread acceptance.

In fact, the recent trend is towards providing QoS and integrated services at the IP level, which minimizes the importance of the underlying transmission platform. This, combined with the emergence of the less complex and less expensive Gigabit Ethernet, reduces the attractiveness of ATM as a backbone network technology. The fact that all the network traffic at K-State is frame-based Ethernet packets further diminishes the attractiveness of ATM. It is inefficient to take a 1500 byte Ethernet packet, split it into many 55 byte ATM cells, transmit it across the backbone, then reassemble them on the receiving end back into a 1500 byte Ethernet packet.

While the fault tolerance built into ATM is very attractive, the above arguments along with the fact that we can do much more on the backbone with Gigabit Ethernet for less than half the cost of ATM leads me to recommend a Gigabit Ethernet backbone technology.

Implementation of Initial Phase

1. Install a core Gigabit network of two Cisco Catalyst 5500 switches located at the Power Plant (PP5500) and Hale Library machine room (Hale5509), and a Cisco Catalyst 4000 switch in College Court (CC4000). Since the College Court switch only serves one building, it needs neither the flexibility nor the routing capability of the larger Catalyst 5500 series.
2. Connect the following buildings to PP5500 at 100 Mbps: Ackert, Bushnell, Calvin, Cardwell, Chem/Biochem, Dickens, Durland, East Stadium, Fairchild, Holtz, King, Nichols, Power Plant (Telecom dial-up IP), Seaton, Throckmorton, Umberger, Waters, and Willard.
3. Move the Catalyst 5000 from Hale Library to Throckmorton to serve as a 100 Mbps fiber backbone switch. Four additional Cisco 2924 switches will need to be purchased by Throckmorton residents if this is to happen. We will leave one existing 2924 switch there, and move the other to Anderson Hall so it can connect to PP5500 at 100 Mbps. The Catalyst 3000 in Anderson Hall will move to Dykstra to alleviate internal problems resulting from a loaded shared 10 Mbps network.
4. 100 BaseTX interface from Anderson 3000 switch gets moved to Bluemont 3000 to connect Bluemont to PP5500 at 100 Mbps.
5. Connect the following buildings to PP5500 at 10 Mbps (Qty. 11): Ahearn, Burt, Dykstra, GMRL, Holton, Military Science, Shellenberger, Thompson, and West Hall AGS+.
6. Connect the following to the Hale5509: enterprise servers in Hale CNS machine room at 10 and 100 Mbps; Hale Library Cisco Catalyst 3000 switches at 100 Mbps; CNS staff networks.
7. Install 100 Mbps 10/100BaseTX SBUS cards in mesa and horizon, two of the Human Resources servers (Sun Enterprise 1000E servers).
8. Remove Cisco 2511 router serving as a firewall in front of the H.R. servers. Connect all H.R. servers to Hale5509 at 100 Mbps and set up access lists on Hale5509 that match the filtering provided by the retired Cisco 2511 router.
9. Connect DCE servers, switches, and hubs to the CC4000.
10. Move Cisco 7000 router from the Power Plant to the VetMed complex to replace the outdated AGS+ router. Connect the 7000 back to the core PP5500 switch with two 100 Mbps Ethernet links and configure them as a Fast EtherChannel to get full duplex, fault tolerant, load balancing 200 Mbps performance. Connect the three VetMed buildings (VCS, VMS, VMT) to the 7000 at 100 Mbps. Other north campus buildings will connect to the 7000 with 10 Mbps Ethernet (Bramlage, Call, Edwards, Football, Jardine, Pittman, Recreation Complex, and Weber).
11. Connect the Cisco 7513 in the Power Plant to the PP5500 with two 100 Mbps Ethernets in a Fast EtherChannel configuration providing full duplex, fault tolerant, load balancing 200 Mbps performance. Connect the following buildings to the 7513 with 10 Mbps Ethernet (Qty 9, 6 more ports available): Beech Art Museum, Galitia, Goodnow, Lafene, Leasure, Kramer, Marlatt, McCain, and Power Plant (Facilities offices, Photo Services).
12. Install eight new serial interfaces in the Cisco 7513 router in the Power Plant. Connect Salina, Konza Prairie, and the Manufacturing Learning Center to these interfaces.
13. Connect the existing Cisco 5505 switch in the Foundation Center to PP5500 with two 100 Mbps Ethernets in a Fast EtherChannel configuration providing full duplex, fault tolerant, load balancing 200 Mbps performance.
14. Connect the existing Cisco AGS+ router in West Hall to the PP5500 with 10 Mbps Ethernet (Boyd, Derby, Ford, Haymaker, Moore, Putnam, Smurthwaite, Van Zile, and West).
15. Use the following freed-up switches for other buildings: Cisco 2900 (14 10/100BaseTX ports) from the Power Plant; Two Cisco 5000 switches (24 10BaseT, 12 100BaseTX, 12 100BaseFX) from Hale Library; perhaps Cisco 2901 (14 10/100BaseTX) from College Court; Cisco 2511 that served as the H.R. firewall; Cisco Catalyst 3000 from Anderson Hall with one 100BaseTX module.

Estimated Cost for Initial Phase

Internet2 Project Needs

The new backbone clearly meets UCAID's network upgrade expectations of its member institutions. Also, buildings with researchers specified in the NSF HPC (vBNS) Connection grant need at least a 100 Mbps connection to the core. Cardwell, Durland, and Nichols already have 100 Mbps connections. Chem/Biochem, Throckmorton, and Willard are still at 10 Mbps, will connect at 100 Mbps once the new core network is in place.

We also need to provide at least switched 10 Mbps Ethernet (preferably switched 100 Mbps) to the computers used by the Internet2 researchers within their buildings. In some cases, new CAT5 cable will need to be run by Telecommunications. We will work with each on a case-by-case basis.

Building Status at the End of the Project
1. New Gigabit Ethernet backbone nodes:

• Power Plant
• Hale Library
• College Court

• Cardwell
• Durland
• Hale Library
• CNS servers in Hale
• KSU Foundation Center
• Nichols
• Power Plant (Telecommunications dial-up IP modem pool)
• Seaton
• Umberger
• Waters

• Ackert
• Anderson
• Bluemont
• Bushnell
• Calvin
• Chem/Biochem
• Dickens
• East Stadium
• Fairchild
• Holtz
• King
• Throckmorton
• Veterinary Medicine:
• Trotter
• Veterinary Medical Sciences
• Veterinary Clinical Sciences
• Willard

• Dole
• Eisenhower
• Justin
• Kedzie
• Ward

Other Options/Issues

• From 4 to 9 spare 100BaseFX (multimode fiber) interfaces will be available on the PP5500, depending on how many are utilized in this first phase, to connect more buildings at 100Mbps in the future. It will also have 5 spare Gigabit Ethernet ports and one 10BaseFL interface (10 Mbps fiber). It will also have 2 to 4 slots available in the chassis for future expansion.
• The Hale 5509 will have two spare slots for expansion, 7 spare gigabit Ethernet ports for connecting enterprise servers, and at least 10 spare 10/100BaseTX ports for adding more servers or workstations.
• Gigabit Ethernet to Dole? KFC? CIS in Nichols? A Cisco Catalyst 4003 is a good device for this purpose - modular with three high density slots. A configuration with 2 Gigabit Ethernet ports and 32 10/100 BaseTX ports costs $11,540. Another good device is a fixed-configuration Cisco 2948GX with 2 Gigabit Ethernet ports and 48 10/100 BaseTX ports for about $6,300.
• Buildings that would benefit from a 100 Mbps connection as well as switch technology, but require the purchase of the switch, are: Eisenhower, Ward, Justin, Kedzie, and Dole. Eisenhower is best suited for a switch due to a large number of users and hubs in the building. Justin and Dole need significant re-wiring to benefit from a switch due to large thin or thick coax Ethernet segments.
• The Union has an old Kalpana switch that is very similar to the Cisco Catalyst 3000, but has been end-of-life'd by Cisco and cannot be certified to be y2k compliant.
• Same is true for the Cisco AGS+ router in West Hall - it has been end-of-life'd by Cisco and must be replaced within the next year, before the year 2000.
• The Cisco 7000 router in the VetMed complex serving the north campus buildings will be full - it cannot have any more connections or modules added to it.
• In late summer, 1999, the Salina campus will be connected to the Manhattan campus with a T3 circuit (45 Mbps). We will need an additional OC-3 ATM interface in the Cisco 7513 router in the Power Plant at $5,600. We will also need to install in Salina a device capable of routing and an OC-3 ATM module. One option is to replace the Cisco 7000 router in the Vet Med complex with a new 5500 Gigabit switch to extend the high speed core network, then move the 7000 to the Salina campus.

If you have any comments, questions, or recommendations for this plan, please contact:

• Harvard Townsend, Interim Director of Computing and Network Services, 532-6311, harv@ksu.edu
• Richard Becker, Network Manager for Computing and Network Services, 532-4909, rlb@ksu.edu
• Dr. Beth Unger, Vice Provost for Academic Services and Technology (VPAST) and Dean of Continuing Education, 532-6520, beth@ksu.edu

Kansas State University | University Committees | CITAC