Required Skills & Experience
• Hands-on experience executing network changes in production environments, including during live maintenance windows
• Strong CLI experience on Cisco routers and switches (IOS-XE, IOS-XR, or Nexus)
• Experience performing and validating software upgrades or configuration changes on network devices
• Ability to read, interpret, and precisely execute Method of Procedures (MOPs)
• Proven ability to troubleshoot issues in real time under tight maintenance window constraints
• Strong understanding of network fundamentals: routing protocols, traffic flow, failover behavior, and device reload sequences
• Comfortable working an overnight shift schedule with consistent reliability
Nice to Have Skills & Experience
• Experience with multi-hop software upgrade paths and intermediate code validation steps
• Familiarity with console access methods and troubleshooting console connectivity issues (CSM/out-of-band access)
• Experience with Cisco Nexus platforms and NX-OS in addition to IOS-XE or IOS-XR
• Experience in telecom or other high-availability, mission-critical network environments
• Exposure to internal network validation or automation tooling
• Experience working within change control and freeze-window constraints
Job Description
Overview
We are seeking a detail-oriented Network Sustainment Engineer to join our Sustainment team on a night-shift schedule, executing software upgrades across a large-scale, multi-platform network fleet. This role operates within tight maintenance windows to upgrade production network devices — including Cisco IOS-XE, IOS-XR, and Nexus platforms — following approved Method of Procedures (MOPs). The engineer works closely with lead engineers and hands off a clear outcome report to the day team at shift end. A strong ability to execute precisely under time pressure and escalate quickly when issues arise is essential.
Key Responsibilities
• Execute software upgrades on production Cisco network devices (IOS-XE, IOS-XR, and Nexus platforms) during overnight maintenance windows
• Follow multi-hop upgrade paths, navigating intermediate code versions as required by platform upgrade sequences (e.g., two- or three-step code progressions)
• Perform pre-upgrade health checks: validate console access, confirm device reachability, verify traffic state, and confirm MOP readiness before any change begins
• Execute MOPs with precision — command by command — and perform in-MOP validation checks at each gate
• Monitor device behavior during upgrade: reload progress, post-boot configuration integrity, routing adjacency restoration, and traffic resumption
• Perform post-upgrade validation using CLI outputs and internal tooling to confirm the device is fully healthy before closing the maintenance window
• Identify and triage issues encountered mid-window: determine whether to back out, hold, or escalate to a lead engineer
• Execute graceful back-out procedures when upgrades cannot be completed within the maintenance window
• Document all outcomes — successes, failures, back-outs — clearly in the shift handoff for the day remediation team
• Coordinate with the Network Operations Center (NOC) and on-call engineering support as needed during the window
• Flag console access issues, device unreachability, or configuration anomalies encountered prior to upgrade attempts
Day-to-Day Work
• Review assigned device list and MOPs before each shift; confirm scheduled maintenance windows and any market-specific constraints
• Validate console access and device health for each device in the night's queue before the maintenance window opens
• Execute upgrade MOPs sequentially across assigned devices, completing pre-, during-, and post-change validation at each step
• Monitor traffic failover behavior and routing protocol reconvergence following device reloads
• Use CLI outputs and internal tooling to confirm software version, configuration state, and network health post-upgrade
• Capture and document outcomes for each device: upgraded successfully, back-out executed, or failed with details
• Produce a clear end-of-shift handoff report detailing completed upgrades, outstanding failures, and any issues requiring day-team follow-up
• Communicate in real time with lead engineers for escalations, and with the NOC for any service impact during the window