Mangement and Operations of the EBONE Bernhard Stockman February 25, 1993 V 0.3 Abstract This paper defines the EBONE Operations Team, its local and central functionalities and corresponding responsibilities with the goal of making the EBONE a highly stable network in terms of management and operations. 1. Introduction. EBONE is formed by its members as defined in the EBONE Memorandum of Understanding. EBONE is a pan-European international networking service providing international backbone service and intercontinental neutral access service to its members. Connectivity will be offered between EBONE member networks and the global Internet. The EBONE Mangement Committee (EMC), nominated by the EBONE members, has the overall responsibility for the EBONE. The EMC nominates the EBONE General Manager (EGM) having the daily administrative responsibility of the EBONE service. The EBONE Action Team (EAT) has the responsibility for the engineering and development of the EBONE. The EBONE Action Team consists of technical representatives nominated by the EBONE members. The EBONE Operations Team (EOT) has the responsibility of the management and operations of the installed EBONE. This document describes the EBONE Operations Team. For more information on the EBONE contact the EBONE Secretariat. (E-mail: ebone@rare.nl, tel: +31 20 639 1131) 2. EBONE Operations Team 2.1 Definitions and Responsibilites Each site installing an EBONE Boundary System (EBS) according to the EBS Technical Specification and the EBONE-92 Implementation plan is referred to as an EBS site. Each site having installed an EBS must have exactly one designated EBS manager. It is recommended that a deputy EBS manager be designated. In order to coordinate the management of EBONE an EBONE Network Operations Center (NOC) is established as defined below. The management and operations of the EBONE is undertaken by the EBONE Operations Team (EOT). The EOT is formed of the set of EBS managers and the EBONE NOC. To be able to undertake the task of coordination and fullfil other defined responsibilities priviliged access to each EBS shall be given to that EBS manager and to the EBONE NOC. EBONE kernel configurations may not be changed by any other parties or persons than the EBONE Operations Team as defined above. Changes can only happen - In normal cases after a request for a configuration change issued by EAT and sent to EOT for implementation. - In an emergency situation by EOT after a trouble ticket have been issued to the relevant sysfail lists describing the emergency and the needed action. With EBONE configuration is here ment the EBONE core line and EBS installations, traffic forwarding configurations such as addressing and routing configurations, management configurations and other basic configurable elements of the EBONE kernel. The EBS manager and the NOC will liaison closely about the operational state of the EBS. The NOC shall respect the executive authority of EBS sites carefully. The EBS manager shall perform actions requested by the NOC in a timely manner. 3. The EBONE Network Operation Center (EBONE NOC). 3.1 Specification. The EBONE Network Operations Center has the overall responsibility for the coordination of management and operation of the EBONE. The NOC will coordinate all EBONE operational activities of the EBS sites. To fulfill this task necessary monitoring equipment and staff shall be installed at the site providing the EBONE NOC. The EBONE NOC personnel shall consist of the EBONE NOC manager and EBONE NOC personnel. Sufficient personnel resources shall be employed to fulfill the EBONE NOC responsibilities. For a detailed job description of the EBONE NOC manager see appendix A. The EBONE NOC reports to the EAT and EOT. 3.2 Specification of responsibilities. The EBONE NOC is responsible for that the EBONE backbone behaves according to the EBONE specification and is reliable. The NOC is responsible for optimal traffic flow on EBONE. The NOC is responsible for fault reporting, isolation and repair. The EBONE NOC has the overall responsibility for changes in the EBONE kernel configuration. Any problems in undertaking these responsibilites for technical reason shall be referred to the EAT. Any problems in undertaking these responsibilites for administrative reasons shall be referred to the EMC, specifically as represented by the EGM 3.3 Specification of EBONE NOC tasks 3.3.1 Mangement and Operations of the installed equipment Normal daily operations such as - Monitoring of EBONE equipment installed at the EBONE NOC, - Monitoring of EBONE kernel reliability and performance, 3.3.2 Mangement and Operations of the IP service. The management and operations of the IP service includes - Monitoring of routing according to the routing plan, - ensuring correct configurations are installed in all EBS routers 3.3.3 Fault management - Issuing trouble tickets to relevant sysfail-lists in case of faults in EBONE - Initiation and coordinating fault repair, - Keeping track of fault conditions, - Reporting status of faults and problems to interested parties, - Maintenance of logs of fault reports. 3.3.4 Operational support for EBS installations and other connected organizations. The EBONE NOC shall collaborate with the EBS managers to maintain the EBONE services. A list of EBS managers shall be maintained for easy and fast contact with the EBS installations. RBS operators shall have the possibility of contact the EBONE NOC directly when there is no operators available at the relevant EBS site. The EBONE NOC shall be possible to contact via email, phone and fax. It is recommended that the EBONE NOC has 7*24 operational coverage to which the designated EBS operators may report faults. Fault resolution will be guaranteed during local office hours and a reasonable effort will be made at other times. 3.3.4 Monthly EBONE reports. The EBONE NOC will have the responsibility of producing monthly reports and to distribute these reports to each of the EBONE member organizations. It is also recommended that these reports are being distributed to other interested parties. The reports shall also be made generally available using "anonymous ftp". The monthly reports shall contain at least: - Total input and output byte and packet at each interface for each EBS. - If possible per protocol and per application byte and packet shown both as totals and as percentages. - Availability statistics for each EBONE core line and each EBS installation. - Other significant events that affect the EBONE core. The EBONE NOC shall receive montly reports from each EBS site no later then two weeks after change of month to be able to inlcude these in the EBONE monthly report. The NOC shall coordinate statistics gathering with the RIPE NCC in order to prevent redundant collection of statistics and to ensure that the collected statistics are useful outside the scope of EBONE as well. 3.3.5 EBONE development and service enhancements The EBONE Network Operation Center shall in close collaboration with the EMC and the EAT recommend, review and supervise changes and additions to the EBONE kernel, regional and intercontinental connections. For this reason the EBONE Network Operation Center will have regular meetings with the EBONE Operations Team and the EBONE Action Team. 4. EBONE Boundary System Sites 4.1 Definition Each site installing an EBONE Boundary System (EBS) according to the EBS Technical Specification and the EBONE-92 Implementation plan is referred to as an EBS site. Each site having installed an EBS must have exactly one designated EBS manager. It is recommended that a deputy EBS manager be designated. 4.2 Responsibilities The task of the EBS manager is to assure the integrity and the functionality of the EBONE defined services. The EBS manager is responsible for local the operation of his EBS installation and acts as an contact point between the EBONE NOC and the EBS installation and as a contact point between the EBS installation and regional connections to that EBS. 4.3 Specification of tasks 4.3.1 Mangement and Operations of the EBONE. The EBS manager shall: - Monitor load, response times and services interruptions on the backbone and on regional connections to the backbone - See to that the services are maintained according to the EBONE service requirements. - Assist the EBONE NOC in resolving problems. - Advise the EBONE NOC on needed technical changes. - See to it that correct configurations are being installed in the EBS. - Maintain operational contacts with the regional networks connected through his EBS. - Provide the EBONE NOC with EBS montly report no later than two weeks after change of month for inclusion in the EBONE monthly report. 4.3.2 Collaboration with other EBS installations The EBS manager shall collaborate with other EBS managers as needed to resolve problems. It shall be possible to contact each EBS manager via phone, fax and email during local office hours. It is recommended that the EBS manager establishes a clear, documented contact and escalation procedure for operational problems. To the extent possible this procedure should also cover non office-hours. 5. EBONE Regional Boundary System responsibilities. 5.1 Specification The EBONE Regional Boundary System is a installation directly connected to one or more of the EBS's. The RBS connects regional networks to the EBONE. 5.2 Responsibilities The RBS installation shall have a nominated contact point for problem resolution. When a problem is discovered at a RBS site which is believed originate in the EBONE kernel the relevant EBS site should be contacted. If this does not succeed the EBONE NOC shall be Regional networks have the responsibility of see to that only traffic from EBONE connected sites are injected into the EBONE. Problems discovered by users in a regional networks should be reported to relevant operators within that network. Problem reporting procedures should ensure that user complaints are not being forwarded directly to the EBS manager or to the EBONE NOC. 6. Problem resolving procedures. To aid the EBONE Operations Team in the resolving of problems in the EBONE, a list of contact persons for the EBONE Operations Team shall be maintained by the EBONE NOC. This list shall only be known to the EBONE Operations Team and nominated responsible persons at RBS installations. 6.1 Problem detected in the EBONE kernel. The EBONE NOC is informed as well as all EBS sites. The NOC is then responsible to coordinate problem resolution and inform interested parties as required. If the EBONE NOC is not available the EBS site first detecting the problem will coordinate problem resolution and inform interested parties as required until the NOC becomes available. This EBS site will state clearly that it coordinates the problem until it can pass the responsibility to the NOC. When the EBONE NOC have been informed and have taken the responsiblity for coordination of fault isolation and correction the tasks described under 3.3.3 above shall be undertaken by the EBONE NOC. Depending on the nature of the problem corrective actions should be immediately started by either the EBS site detecting the problem or by the EBONE NOC. In case of link failures EBS sites should 1) inform the NOC 2) contact the peer EBS sites and cooperate in determining the cause of the failure 3) inform the carriers (or the MUX NOC) as required It is essential to coordinate with the peer EBS site before contacting the carriers. If the peer EBS site cannot be contacted the EBS site and the NOC will try to diagnose the state of the EBS at the remote site before initiating further action. The NOC and EBS sites shall liaison closely and frequently about any actions taken during resolution of failures. 6.2 EBONE related problems detected at a RBS site. The RBS site shall first try to verify that the detected problem is related to the EBONE. If so the relevant EBS site shall be contacted who then will initiate corrective actions as described above. 6.3 EBONE related problems detected within a regional network. Normally users within a regional network shall contact the relevant operational responsible persons within that network. These operators may then initiate corrective actions according to 6.2 above. 6.4 Actions when a problem is resolved. Once a failure is resolved the NOC shall inform all parties that have previously been made aware of the failure about the resolution. 6.5 Problem resolution flow char. As presented in Prague and to be included here.