Mangement and Operations of the EBONE Bernhard Stockman April 13, 1992. V 0.2 Abstract This paper defines the EBONE Operations Team, its local and central functionalities and corresponding responsibilities with the goal of making the EBONE a highly stable network in terms of management and operations. 1. Introduction. EBONE is an European international network backbone connecting network service providers. Connectivity will be offered between these networks, to the United States and to the International Internet. The ECCO Mangement Committee has the overall responsibility for the EBONE. To fulfill this task two subgroups are being defined. The EBONE Action Team (EAT) which have the responsibility for the engineering and development of the EBONE. The EBONE Operations Team (EOT) which have the responsibility of the management and operations of the installed EBONE. This document describes the EBONE Operations Team. For more information on the EBONE contact the EBONE Secretariat. (E-mail: ebone@rare.nl, tel: +31 20 639 1131) 2. EBONE Operations Team 2.1 Definitions Each site installing an EBONE Boundary System (EBS) according to the EBS Technical Specification and the EBONE-92 Implementation plan is referred to as an EBS site. Each site having installed an EBS must have exactly one designated EBS manager. It is recommended that a deputy EBS manager be designated. In order to coordinate the management of EBONE a EBONE Network Operations Center (NOC) will be established as further specified below. The management and operations of the EBONE is undertaken by the EBONE Operations Team (EOT). The EOT is formed of the set of EBS managers and the EBONE NOC. 2.1 Responsibilities EBONE kernel configurations may not be changed by any other parties or persons than the EBONE Operations Team as defined above. Configuration is defined as basic line and EBS installations, traffic forwarding configurations such as addressing and routing, management configurations or other basic configurable elements of the EBONE kernel. 2.2 Division of Responsibilities and Authority Each EBS site will clearly define which actions the NOC can perform on the EBS under which circumstances (maintenance, fault conditions). The EBS manager and the NOC will liaison closely about the operational state of the EBS and any configuration changes. The NOC shall respect the executive authority of EBS sites carefully. The EBS manager shall perform actions requested by the NOC in a timely manner. In case an EBS malfunction considerably harms EBONE operations as a whole, the NOC is not authorized to take remedial action on the EBS, the NOC may isolate the EBS concerned from the rest of EBONE. 3. The EBONE Network Operation Center (EBONE NOC). 3.1 Specification. The EBONE Network Operations Center has the overall responsibility for the coordination of management and operation of the EBONE. The NOC will coordinate all EBONE operational activities of the EBS sites. To fulfill this task necessary monitoring equipment and staff shall be installed at the site providing the EBONE NOC. The EBONE NOC personnel shall consist of the EBONE NOC manager and EBONE NOC personnel. Sufficient personnel resources shall be employed to fulfill the EBONE NOC responsibilities. For a detailed job description of the EBONE NOC manager see appendix A. The EBONE NOC reports to the EAT and EOT. 3.2 Specification of responsibilities. The EBONE NOC is responsible for that the EBONE backbone behaves according to the EBONE specification and is reliable. The NOC is responsible for optimal traffic flow on EBONE. The NOC is responsible for fault reporting, isolation and repair. The EBONE NOC has the overall responsibility for changes in the EBONE kernel configuration. 3.3 Specification of EBONE NOC tasks 3.3.1 Mangement and Operations of the backbone. The management and operations of the backbone includes - Monitoring of EBONE equipment installed at the EBONE NOC, - Monitoring of EBONE kernel reliability and performance, - Keeping track of fault conditions, - Initiation and coordinating fault repair, - Reporting faults and problems to interested parties, - Maintenance of logs of fault reports. 3.3.2 Mangement and Operations of the IP service. The management and operations of the IP service includes - Monitoring of routing according to the routing plan, - ensuring correct configurations are installed in all EBS routers, 3.3.3 Operational support for EBS installations and other connected organizations. The EBONE NOC shall collaborate with the EBS managers to maintain the EBONE services. A list of EBS managers shall be maintained for easy and fast contact with the EBS installations. RBS operators shall have the possibility of contact the EBONE Operation Center directly when there is no operators available at the relevant EBS site. The EBONE Operation Center shall be possible to contact via email, phone and fax. It is recommended that the EBONE Network Operation Center has 7*24 operational coverage to which the designated EBS operators may report faults. Fault resolution will be guaranteed during local office hours and a reasonable effort will be made at other times. 3.3.4 Monthly EBONE reports. The EBONE Network Operation Center will have the responsibility of producing monthly reports and to distribute these reports to each of the EBONE contributing organizations. It is also recommended that these reports are being distributed to other interested organizations. The reports shall also be made generally available using "anonymous ftp". The monthly reports shall contain at least: - Total input and output byte and packet at each interface for each EBS. - If possible per protocol and per application byte and packet shown both as totals and as percentages. - Availability statistics for each EBONE core line and each EBS installation. - Other significant events that affect the EBONE core. The NOC shall coordinate statistics gathering with the RIPE NCC in order to prevent redundant collection of statistics and to ensure that the collected statistics are useful outside the scope of EBONE as well. 3.3.5 EBONE development and service enhancements The EBONE Network Operation Center shall in close collaboration with the ECCO EXEC and the EAT: - Ensure a smooth phase over from current European network infrastructures to the EBONE-92 to minimize performance degradation for involved organizations during this period. - Coordinate the installation and maintenance of EBS configurations - Recommend, review and supervise changes and additions to the EBONE kernel, regional and intercontinental connections. For this reason the EBONE Network Operation Center will have regular meetings with the EBONE Operations Team and the EBONE Action Team for the installation of the EBONE-92 and future implementation of changes in the EBONE kernel configuration. 4. EBONE Boundary System Sites 4.1 Definition Each site installing an EBONE Boundary System (EBS) according to the EBS Technical Specification and the EBONE-92 Implementation plan is referred to as an EBS site. Each site having installed an EBS must have exactly one designated EBS manager. It is recommended that a deputy EBS manager be designated. 4.2 Responsibilities The task of the EBS manager is to assure the integrity and the functionality of the EBONE defined services. The EBS manager is responsible for local the operation of his EBS installation and acts as an contact point between the EBONE NOC and the EBS installation and as a contact point between the EBS installation and regional connections to that EBS. 4.3 Specification of tasks 4.3.1 Mangement and Operations of the EBONE. The EBS manager shall: - Monitor load, response times and services interruptions on the backbone and on regional connections to the backbone - See to that the services are maintained according to the EBONE service requirements. - Assist the EBONE NOC in resolving problems. - Advise the EBONE NOC on needed technical changes. - See to it that correct configurations are being installed in the EBS. - Maintain operational contacts with the regional networks connected through his EBS. 4.3.2 Collaboration with other EBS installations The EBS manager shall collaborate with other EBS managers as need to resolve problems. It shall be possible to contact each EBS manager via phone, fax and email during local office hours. It is recommended that the EBS manager establishes a clear, documented contact and escalation procedure for operational problems. To the extent possible this procedure should also cover non office-hours. 5. EBONE Regional Boundary System responsibilities. 5.1 Specification The EBONE Regional Boundary System is a installation directly connected to one or more of the EBS's. The RBS connects regional networks to the EBONE. 5.2 Responsibilities The RBS installation shall have a nominated contact point for problem resolution. When a problem is discovered at a RBS site which is believed originate in the EBONE kernel the relevant EBS site should be contacted. If this does not succeed the EBONE NOC shall be Regional networks have the responsibility of see to that only traffic from EBONE connected sites are injected into the EBONE. Problems discovered by users in a regional networks should be reported to relevant operators within that network. Problem reporting procedures should ensure that user complaints are not being forwarded directly to the EBS manager or to the EBONE NOC. 6. Problem resolving procedures. To aid the EBONE Operations Team in the resolving of problems in the EBONE a list of contact persons for the EBONE Operations Team shall be maintained by the EBONE NOC. This list shall only be known to the EBONE Operations Team and nominated responsible persons at RBS installations. 6.1 Problem detected in the EBONE kernel. The EBONE NOC is informed as well as all EBS sites. The NOC is then responsible to coordinate problem resolution and inform interested parties as required. If the EBONE NOC is not available the EBS site first detecting the problem will coordinate problem resolution and inform interested parties as required until the NOC becomes available. This EBS site will state clearly that it coordinates the problem until it can pass the responsibility to the NOC. Depending on the nature of the problem corrective actions should be immediately started by either the EBS site detecting the problem of by the EBONE NOC. In case of link failures EBS sites should 1) inform the NOC 2) contact the peer EBS sites and cooperate in determining the cause of the failure 3) inform the carriers (or the MUX NOC) as required It is essential to coordinate with the peer EBS site before contacting the carriers. If the peer EBS site cannot be contacted the EBS site and the NOC will try to diagnose the state of the EBS at that site remotely before initiating further action. The NOC and EBS sites shall liaison closely and frequently about any actions taken during resolution of failures. 6.2 EBONE related problems detected at a RBS site. The RBS site shall first try to verify that the detected problem is related to the EBONE. If so the relevant EBS site shall be contacted who then will initiate corrective actions as described above. 6.3 EBONE related problems detected within a regional network. Normally users within a regional network shall contact the relevant operational responsible persons within that network. These operators may then initiate corrective actions according to 6.2 above. 6.4 Actions when a problem is resolved. Once a failure is resolved the NOC shall inform all parties that have previously been made aware of the failure about the resolution. APPENDIX Job description for the EBONE Operations Manager (The Skipper). A.1 The EBONE Operations Manager shall: Act as the main liaison between the EBONE NOC and the EBONE Operations Team and the EBONE Actions Team. See to that regularly reports about the activities performed inside the EBONE NOC are distributed to interested parties. Take the lead in the coordination of the activities of the EBONE Operations Team. Act as main interface and reference point between each EBS responsible and the EBONE NOC. Chair EBONE Operations Team meetings and discussions. Participate in the activities of RIPE and act as the EBONE NOC interface to the RIPE NCC. Insure proper coordination among EBONE NOC and other networks not directly participating in the EBONE initiative. A.2 In particular the EBONE Operations Manager has the responsibility of: Insure that the tasks defined in section 3.3 of this document are properly executed and, when applicable, the required output is produced. Assign and distribute tasks to the EBONE NOC staff and coordinate the activities inside the EBONE NOC. Coordinate all the necessary technical changes in the EBS's and insure that the changes are forwarded to all the interested parties outside the EBONE. A.3 Profile of the EBONE Operations Manager. Scientific degree or an equivalent diploma. At least 3 years work experience in the international academic network world as service provider. An excellent knowledge of the TCP/IP protocol suite. A good knowledge of IP router management and installation procedures (a specific skill on CISCO equipment is desirable). A basic knowledge of statistics. Previous experience of technical management and people coordination. A.4 The EBONE Operations Manager shall also: Enjoy working in an international environment and have no problem in communicating and cooperating in a team. Be available for travel in Europe and the US. A very good knowledge of the English language is mandatory for this position. The knowledge of a second European major language is highly desirable.