In this article I explain the troubleshooting of intermittent internet and ignite TV issues from Rogers Communications.
Using Rogers XB6 modem
Over the past year and a half to 2 years, my Rogers experience has not been ideal, in fact less than ideal with many network problems causing me to lose internet connectivity on a frequent basis. Outages are now back to a daily occurrence and becoming a running joke in my household. At times, speed tests are horrid - bad jitter, upload/download speeds, etc. Basically an unacceptable MTF (mean time to failure) rate. In the past, and right after getting Rogers installed, I have had multiple technicians come by to test lines/etc. - each claiming that there are issues “noise” on the lines. I have had a new line run to the tap/node, new terminations, etc. After which they claim everything is fixed. Each of these have worked for a short while, then issues return. Basically, there is a tap/node side issue that causes the network to be very unstable and it needs to be formally resolved.
I have attempted to get this fixed on numerous occasions, but end up having to give up as I don’t have the energy or time to get Rogers to get their act together. However, it has gotten to the point where the network is so bad that I can’t even watch a single movie or hockey game without reboots!! As such, I am back dedicating time to try to troubleshoot and convince Rogers that they have a significant issue with their service and try to get a resolution. This is a log of just some of that interaction.
Background:
Previously, I was running in “bridge” mode, effectively bypassing the Bell and then Rogers routers/firewalls with no issues on Bell Fibe but Rogers has been an issue since day 1 and for over a couple years both in Bridge mode and when using their modem/firewall Rogers has been horrible. With Bell Fibe, I was running my own Fortigate 60d Wifi 4th gen firewall/router to manage access, monitor links, etc. With two 48 port 1GByte switches all running Cat 6 cabling. Everything ran smoothly on Bell Fibe in this configuration with few if any outages. Everything collapsed when I went to Rogers - Here is the FortiView - System Events - "interface status changed" showing that the Rogers modem kept dropping - even in bridge mode. This is over 1 1/2 years ago and I have been trying to get any reliability out of Rogers since but it never gets totally fixed!:
Unfortunately, I lost the Fortigate during a power outage recently and decided to give the Rogers Gateway Ignite Modem/firewall a try and take the modem out of bridge mode. Oh boy! I did not think things could get much worse - but boy was I wrong!!
The router is rack mounted and the IPTV Ignite boxes (2) are LAN connected. I am running several devices on my network (wired and wireless) and when you consider the Wifi enabled home automation switches - in the 100’s. But these are supported by multiple Wifi routers configured to manage their own subnets.
I also have a web server running on a NAS - but for security reasons it runs in a separate virtual machine (so that I do not have to expose the NAS to the internet). All internet facing services/ports are scrutinized and/or disabled. For security reasons, I turn off upnp on the Rogers modem and do only minor port forwarding. I now rely on the Rogers Advanced Security to protect against unwanted access requests, etc. but the network itself is totally unreliable.
The Issue:
The modem tends to drop its connection intermittently. Ramping up over time - currently restarting (soft resets) several times a day. No real rhyme or reason - however I am not able to see failure logs (or wireshark) and have to rely on Rogers support staff but it is extremely frustrating when they tend to claim things are working when they obviously are not.
I have spent many hours (too many) with support staff of varying degrees of knowledge - most able to do the basic testing before putting in a ticket for a technician visit. The most recent issue resulted in yet another Technician visit that ended prematurely with the technician cancelling “en route” claiming that the issue was wide spread and that there was a “congestion” ticket being raised to deal with issues at the “node” level. The challenge with congestion issues is that while it could, it would not likely result in modem disconnects?? More likely congestion would contribute to performance issues. There has to be a more fundamental issue like noise on the line?
While I suspect that fixing the congestion issue at the node will help, I am not sure if it will finally fix the root cause. As such, I must rule out any other potential causes for the issue as I cannot wait the “months” (quoted by the Rogers help desk staff) to fix the congestion issue. Currently, I have no ignite TV picture but the internet is connected and the IPTV devices are connecting and providing a "guide" but no picture.
Why Are Other Users Not Complaining Constantly?
Sadly there are a few reasons why this is not being taken too seriously by the Rogers team? Rogers relies on users not noticing the drops as they tend to be fairly short (1-2 min in length). Rogers by default uses buffering and "herds" their users users to connect to their wifi radios on their routers. If users are not peer-peer connected at the time of the outage, it can go unnoticed or seem like a temporary "glitch". For example web browsing would likely not be too noticeably impacted as it is usually more static with polling. Also the IPTV boxes when Wifi connected can potentially buffer ahead to “smooth out” or “ride out” a short term “soft restart” outage on their router. The symptoms appear to point to a "soft drop" and this is what happens when the router has trouble connecting to the node. For me - I am using many peer-peer connections including hard wiring the IPTV boxes. Thereby immediately noticing even these shorter “soft” resets / outages but I no longer have the luxury of my Fortigate WAN interface logs. In general - Rogers keeps telling their users that things are fixed and it must be their specific issue and unfortunately most end users are not technical enough to challenge these claims - I am not one of those users.
The "Non-Scientific" Resolution / Proof:
Based on testing to date my suspicion is that the gateway/router is having issues connecting to the node forcing a "soft reset" but it may be failing for a number of reasons:
- There are issues between the router and the node - the router is loosing connection to the node (my belief). This can be due to an unstable infrastructure - a bad tap, cabling, noise, grounding, other??
- The modem/router is under powered. Meaning that the power supply is not able to deal with peaks in load? (Claimed by Rogers support staff?)
- There are packets on the network that are confusing the modem/router causing it to complete a soft restart? (Claimed by Rogers staff?)
- EMI (electromagnetic interference) causing the router to complete a soft restart? Even when not using a radio (wifi)?? (Surprisingly also claimed by Rogers support staff?)
All of these are theories (mine for the former and the latter three claimed by the Rogers help desk agents) but for now I will try a test to prove that the latter theories are not valid - leaving only the fact that it must be an issue with their infrastructure - likely the link from the modem to the tap/node/hubs/distribution gear, etc. Feeding to the frustration, is that when I called recently for an update on the "node repair ticket" claimed by the Rogers technician who failed to show up, the Rogers help desk agent claimed there was no “node” or network side issues in the area and that services were all fine and that it had to be an isolated issue at my end/location, including claiming crazy theories like EMI or radio interference from electrical panels, etc. wanting to replace the modem and IPTV boxes?? To prove this is not the case, I reluctantly tested the power supply and EMI theories theory by moving the modem and lightening the load on the modem/router by disabling both the wifi radios (2.4 and 5G) since I am not relying on them for anything specific and reboot the router (and the tv boxes). Sure enough the reboot cleared my issue - both ignite tv boxes came back and are working as expected. However, this is just an artifact of the restarts as this would also clear any cached networking issues along with routing tables, etc. as well as creating a new connection to the node. I restarted all devices (to clear any residual network caching, arp tables, etc.) and I suspect it will be short lived and will see how things progress. If the modem issue persists - it is very, very likely that it is in fact a connection issue outside the modem (I.e. from the modem back to the node). At least I can be even more confident it remains a Rogers network issue.
With respect to potential network issues causing the modem/router to fail - I have many connected devices on separate wifi routers (with their own address space) and those packets (including icmp from wifi home automation devices) arguably theoritically could cause issues/confusion for the Rogers router causing it to do a soft restart? However without more data (i.e. access to the router logs or wireshark'ing) this is pure speculation? But if this is true - the modem itself would have to have a significant fundamental logical flaw. So I seriously doubt that this is the case.
For now, I will run in this mode with the wifi radios off, trying not to over task the Rogers Ignite router too much and see if I still get intermittent gateway/router drops/resets. While this is not too scientific - if it starts to fail it would explain a lot of the intermittent issues that I am hearing from a variety of Rogers customers - especially in the neighbourhood. And prove that it is more than likely in the outbound side of the router (drops connecting to the node).
Update:
Sure enough within the first hour of light use the router dropped again. The router lost connection taking out the internet and the TVs dropped out while it reconnected. It appears it is not a loading or networking issue on the modem. And since all devices were restarted and routing tables all flushed, this confirmed that it is likely not a packet / routing issue. Therefore we are back to the connection between the router and the node. Unfortunately, this means that I need to rely on Rogers to troubleshoot the line and congestion on the node as it doesn’t appear to be the loading or packet traffic. I am now re-convinced it is not a local network issue but more a back-end issue... and can yet again discount my last interaction with the help desk agent! I am calling Rogers back to explain the situation and see if they change their story?
Eureka - A Small Step Back to Sanity!! - after speaking again with yet another helpdesk agent - who worked with his supervisor - they confirmed/admitted that there is in fact is a serious network issue at the node level on their network that is impacting multiple users in the area/neighborhood. On this call the agent was much more transparent, reviewing the connection logs and confirming that the failure mode is with drops between the modem and the node side of the network. It is sad and bizarre that the previous agent who also had access to all the same information insisted that there was no network issue and that the issue must be isolated at my location! Instead, he insisted that I test things like EMI interferences, etc. (which I totally disagreed with having any potential at being the root cause but I did just to prove a point). Rogers (on this call) formally admitted that the issue is in fact confirmed to be affecting multiple customers in the area (at least I finally have a ticket number for my issue :-0 C177404100). Rogers also claimed / confirmed that a senior Technician team is required to address the root cause - The exact same claim made by the technician who cancelled his visit two days earlier. I am currently on hold waiting for a status update on the root cause issue fix - lets see what they say..... In the meantime, it is sad that you have to argue with a help desk agent going so far as having to prove to them that they have a fundamental network issue?? Most end uses would not be savvy enough to realize that they are being mislead. With the root cause identified, the Supervisor (on this latest call) is attempting to raise an urgent “Maintenance ticket” to have a senior network technician team work on the node side of their network to finally fix the root issue. At my request, a cross reference note is being made on my file so that I do not get yet another help desk agent claiming that is a localized issue and tries to go down more rabbit holes, wasting more time with circular testing / troubleshooting with no way to resolve. Now I am to wait to get status updates from the networking team on their progress to fix the node-side connection issues. Yet another day wasted educating Rogers help desk teams….hopefully I will finally get a more stable and reliable connection!
The Next Day:
Today (the following day) was another day of intermittent outages but now when the ignite IPTV boxes come back they are very sluggish - taking several seconds to call up a guide, change channels, etc. Attempts to watch recordings results in errors. Restarting has no performance improvements. Drop outs continue every hour or several times an hour making watching TV difficult and the internet being totally unreliable. I will be calling tomorrow to attempt to get a status update on progress.
Day 5.000 Update 
:
Just had a call with Rogers help desk who had to escalate the issue as I had no connectivity... As precaution, I enabled upnp for testing just in case it would help... I reset the modem (the networking module) from the UI - no joy. But then I noticed that the modem did not connect properly after the reset and was flashing white - meaning not good. I repowered the modem (hard reset by unplugging it) and got things running for now....
This help desk agent was very friendly and after reviewing the notes on my file understood my frustrations. He also confirmed that the issue is continuing and is on the node side of the network and that they are going to resolve the issue by dispatching a senior network team to resolve (master ticket #7075520). Unfortunately, they are 3 days out from working on it!! So at least 3 more days of wasted time…
Day 5,004 Update:
It is 4 days later and nothing is fixed and not working. So I called Rogers and the help desk has claimed that everything is fine and the area ticket is closed as they tested and there are no issues in the area. This is now officially a joke!
I am escalating once again as the issue is not resolved and I now believe that Rogers does not know how to fix the issue, is incompetent, or is straight up lying to its customers!
They don't know what to do - so at my insistence are dispatching yet another technician to look into the issue. (Ticket: C177609209). Technician is coming immediately (in next 5 hours) to do the assessment.
Technician Visit - Update Ticket# C177609209:
The technician arrived, I briefed him and he put his spectrum analyzer on the line and immediately saw that there is significant noise on the "new" line between the house and the the node/tap side. He also confirmed that I have > 250 recent drops indicating a major problem at the "tap" on the network side. He checked the line into the house (to the modem) and it is clean. The good news is that we are getting close to finding the root of the issues - the bad news is that if this is confirmed - it will require a network team to resolve - again!!! I am getting extremely frustrated with the amount of effort required to get Rogers to admit, find and then ultimately resolve their issue. This issue is years old!
Ok - the technician confirmed that the "tap" is full of noise and is unusable and needs to be replaced. He checked all lines and they are bad (i.e. not just my connection and therefore affecting other users in the area). He also re-checked the line from the tap back to the house and it is clean.
Rogers uses "taps"
This "bad tap" is speculated to have contributed to my issues for over 2 years now that I have been claiming to Rogers and they continually claim that there is no issue!! Unfortunately, even thou it is an easy fix, the technician is not allowed to fix and therefore it is likely going to take a while to get replaced. So in an attempt to make things a bit more useable, the tech is also swapping out my modem to the Gen 2 version. Going from an XB6 to an XB7 modem. He hopes that this modem will better handle the noise on the line until it can get fixed. So I will need to reconfigure my port forwards, etc...
Upgraded to Rogers XB7 modem
I am up and running on the new modem and I am to wait 7 days (hopefully error free) and then call for an update on when the tap is to be replaced. Back into waiting mode....
Performance using the new modem is more stable. Will call next week to get a status update on the tap replacement.
Update - More Than Two Weeks Later
Started to notice more network outages and was preparing to contact Rogers to push on the need to replace the "tap" when all of a sudden at 4:44am (Friday June 8, 2022) I was awoken by the network disconnecting. I got up and began trouble shooting - assuming that it was isolated to our local "tap" and that I would have to contact Rogers in the morning. In the morning I discovered that Rogers had suffered a nationwide outage affecting more than 10 Million customers with no ETA for resolution.
My neighbor approached me stating that he has been living with multiple outages and wanted to know what I had experienced. I explained the issues and he had a technician come out and replace his modem and he also went up the tap and I explained what had occurred. He concurred that the line from the node to the tap was too noisy but that he could not do anything about it and that the "network team" had to look into. He was going to also make a request to resolve.
Update - September 27, 2022 - It is Likely more than just a bad tap!
Lost internet/TV, etc. again today. Been having intermittent issues over the past few weeks but lost everything today. When I called Rogers they claim everything in the area is fine and has to be my house. The also claimed that a network team came out on June 13th (ticket C177404100) and that "all issues are fixed". I could not get the modem to reconnect - i.e. after powering it goes orange, flashing orange, then flashing green and then stuck (meaning it can receive info from Node but cannot establish a reliable connection back). Take a look at the ticket number that Rogers claims to have been the network team fixing the issue!! It is my ticket number from one of my calls!! The helpdesk agent claimed that it must be on my side and that, once again, he wanted to reset my modem to factory defaults and start all over - knowing that his would not result in any improvement in reliability. I kindly refused his offer.
Amazingly, there were 5 (count them 6) rogers trucks on my street when I stepped out after my call to put out the garbage and I ran up to them and said that "they knocked me out" - the technician looked confused :-). These trucks were the infamous Rogers "network" team and they were looking at a major (multiple failures) on the other side of the street. So much for Rogers claiming "everything in your area is fine" I received from the agent!! They looked at the outage map and confirmed that something was, in fact, wrong at my place. After discussion and providing some background, they left to finish what they were doing but they sent another technician over a few minutes later to try to help. Based on the work the team had completed up the street, the speculation is that the signal to noise ratio was changed and he recommended that we try taking the filters off to see I can get reconnected. He took the filter off the line in the junction box at the side of the house but this did not help. I then removed the filter at the back of the modem and I finally could get it to reconnect. The network guy was very candid and said that the network infrastructure for the street was very old and located in an underground conduit that was badly corroded due to flooding and that the removal of the filters may be just a temporary fix. Basically, he said that it was unlikely that things were gonna improve given the state of the infrastructure and that I should consider going back to fiber (;-o). He said to see how things go in the evenings when the network is loaded (i.e. spikes in the signal to noise ratio) and keep the filter in case it is needed to try to resolve issues in the meantime.
Lost connection at least 2 more times in the last 24 hours. Basically, the network is still grossly unstable with no fix in site...
Looks like Rogers is not living up to their recent claims to gain faith from their customer base! They claim that they are committed to improve service/support - but this has just been an example of horrific service/support (in the order of years!) where they are willing to tell clients that things are working when they now explicitly that it is fundamentally unreliable and not working. Effectively, I have wasted endless man hours and service disruptions, etc. only to prove that Roger's is not committed to improving their network services despite their recent renewed claims.
Nov 16, 2022:
Modem dropped and will not reconnect. There was a large snowstorm the past couple days and this likely is affecting noise on the node side of the network. Tried reboot and the signal comes back but dies after about 5 min. I ended up re-installing the noise filter at the modem and this allowed me to keep the signal up for now…
Comments powered by CComment