Issue
A user may experience a reboot loop/loss of connectivity on their Network Management Card 2 (NMC2) device upon upgrading from v5.X.X to v6.X.X. This very specific issue typically results in one or more of the following behaviors:
- A message in the device event log: "TCP/IP stack failure. Module: tfDnsGetMailHost-assertion error line 1266, file(.\treck\source\trresolv.c)" (which is the best way to verify this specific issue)
- A message via the web UI indicating: "You are attempting to access an APC device. The application you are trying to load is incompatible with the current APC OS. Please verify the correct firmware is loaded."
- Loss of connectivity after configuring SMTP settings or where an event occurs that generates an email per the user's notification settings.
Product Line
- Network Management Card 2 - AP9630/AP9630CH, AP9631/AP9631CH, AP9635/35CH
Devices with an embedded Network Management Card 2 include (but are not limited to): 2G Metered/Switched Rack PDUs (AP86XX, AP88XX, AP89XX), Certain Audio/Video Network Management Enabled products.
Environment
- All serial numbers
- AOS module firmware versions v6.0.6, v6.1.1, v6.1.2
- Users attempting to configure SMTP/email
- Users with SMTP/email notifications already set up
- Users utilizing the FQDN for their specified SMTP server
Cause
This is a NMC2 firmware bug that has to do with the NMC2 querying a mail exchange server that is NOT cached by the DNS server a user has configured (meaning the MX record for the SMTP server being used is not cached). Once the mail exchange server is cached on the DNS server, the NMC2 begins to operate as expected.
Refer to the symptoms below and their role in the issue to help identify if this is the issue you're experiencing:
- A message in the device event log: "TCP/IP stack failure. Module: tfDnsGetMailHost-assertion error line 1266, file(.\treck\source\trresolv.c)"
- Recommendation: Review your event.txt file which is where this will be. Instructions in downloading this file and the other log files that may assist are in knowledge base FA156131.
- Recommendation: Review your event.txt file which is where this will be. Instructions in downloading this file and the other log files that may assist are in knowledge base FA156131.
- A message via the web UI indicating: "You are attempting to access an APC device. The application you are trying to load is incompatible with the current APC OS. Please verify the correct firmware is loaded."
- Recommendation: The Network Management Card 2 can display this message for a few different reasons, including when the incorrect application is loaded on the device. Assuming the application version is correct, this can also happen when the NMC2 is in a reboot loop and cannot load the application six (6) times in a row. If the NMC2 is in a reboot loop due to the issue outlined in this knowledge base, you may see this screen as well. The event.txt should be reviewed to see if the aforementioned TCP/IP stack failure message is present. If the web interface is not accessible (which you can try and issue a reboot command via local console, telnet, or SSH), which if generates an email will most likely trigger the TCP/IP stack failure anyway, then you'll want to retrieve the logs using the steps via FTP or logzip command in knowledge base FA156131.
- Recommendation: The Network Management Card 2 can display this message for a few different reasons, including when the incorrect application is loaded on the device. Assuming the application version is correct, this can also happen when the NMC2 is in a reboot loop and cannot load the application six (6) times in a row. If the NMC2 is in a reboot loop due to the issue outlined in this knowledge base, you may see this screen as well. The event.txt should be reviewed to see if the aforementioned TCP/IP stack failure message is present. If the web interface is not accessible (which you can try and issue a reboot command via local console, telnet, or SSH), which if generates an email will most likely trigger the TCP/IP stack failure anyway, then you'll want to retrieve the logs using the steps via FTP or logzip command in knowledge base FA156131.
- Loss of connectivity after configuring SMTP settings or where an event occurs that generates an email per the user's notification settings.
- Recommendation: You may experience this because of or in addition to points 1 and 2. The event.txt should be reviewed and analyzed to see if this issue exists.
Resolution
This issue is resolved in AOS v6.1.3 or higher. Each Network Management Card has an AOS and application file and they cannot be updated independently. Please check https://www.apc.com/tools/download/index.cfm to verify the latest versions available for your device.
For users that are unable to upgrade or if your application does not have AOS 6.1.3 or higher bundled yet, the following options are available:
- Pursue caching the MX Record for your selected SMTP server on the DNS server (or use a DNS server that has the record cached)
- Consider downgrading to v5.1.X firmware for your application while reviewing the information and support for downgrading in knowledge base article FA167693, titled Things To Consider When Upgrading or Downgrading a Network Management Card 2 (NMC2) Device between v5.X.X and v6.X.X.
Published for:APC Belgium
