The main reason for a second back-haul is to offer choice to our customers, and specifically to allow multiple line installations to have an extra level of redundancy. The theory is that, regardless of how bad either carrier is, they will not have the same issues at the same time.
I'm sorry to report that we are now starting to see congestion issues in Be/O2. This has been going on for a while, and so far we have muddled through. The main issue here is that O2 are upgrading their network, which is good, but they are behind the usage, so there are cases of severe congestion happening and it being months before any planned upgrades. We have changed customers back to BT, or waited for the upgrades, but it is starting to be a real problem in some areas now.
I have put off blogging on this to be fair to them and give them a chance to address these issues, but we are not really making any progress on this. So, I'm just reporting how it is so that other ISPs know.
I'm pleased to say that we have really got on top of this sort of issue on the BT side - they are proactive in network upgrades now, and are responding reasonably quickly when there are exceptions. They are considering congestion issues (loss and latency) to be a fault, although it can take some persuading.
Fluidata are not, and are in fact pointing out that their "unlimited" Business grade wholesale services are not in fact unlimited. They are blaming unspecified other "super users" for causing congestion ahead of planned upgrades. Apparently we would have to justify excessive usage (excessive, on an "unlimited" service) if our users were the cause of the problem, rather the victim as in this case. That is somewhat of a concern.
It is a shame. I really thought we were doing a lot better with them and that they would take such issues seriously as a fault rather than washing their hands of them. We had lots of discussions on this early on as we had concerns that the pricing model was not one that scaled well. Whilst we all hate BTs expensive back-haul charging, the fact it is based on usage does present a business model that can scale (we'd just like that charge to be lower in BT's case). It seems that perhaps we were right to have been concerned.
The good news is we hope to have another back-haul carrier on line very soon - they do have a usage based model and are priced so closely to BT back-haul that we will be selling on the same terms as we sell BT 21CN services. We don't have any lines live yet, so cannot yet report on how well they work or how they handle faults or consider congestion issues. We'll be able to report on that after a few months I expect. This will give customers even more choice. We'll make more formal announcements on the A&A pages when we can start taking orders, but it will not be long now.
In the mean time, apparently, according to Fluidata, this is not a proper fault and can wait until April to be fixed (or sooner if they get these other super users to stop using as much). Not impressed.
Update: Slightly concerned at comments that I have got the wrong end of the stick. The above line is either considered faulty and so needs fixing promptly, or it is not considered a fault meaning I haven't got the wrong end of the stick. It should not matter if the person we by a service from buys services from someone else or not - all that matters if that the service works properly and is fixed promptly if it is not working. We don't really ask for much!
The news items on the main page of www.beusergroup.co.uk list known congested exchanges.
ReplyDeleteWorrying that it is spilling over to the wholesale/business services.
DeleteQuite a while ago I did give the list of "usergroup known" congested exchanges to staff in your IRC, they looked up some lines and said that they didn't see any problems when f8lure could see issues to BE retail customers..
DeleteI assumed that BE were giving higher priority to wholesale.. Who knows, maybe the exchanges you have issues with are at a link in the chain where they can't do that.
Interestingly, we aren't seeing similar losses in graphs on our Be/O2 lines (we buy direct from O2 Wholesale rather than Fluidata) - are you sure it isn't a "Fluidata blaming Be/O2 for their own capacity shortfalls" fault?
ReplyDeleteIt is only specific areas - we can discuss where we are seeing problems with you if you like.
DeleteVery disappointing to hear this really, as it was seeming like BE were doing well. I hope they do get on top of it.
ReplyDeleteOur monthly bill on A&A/BE is around £60 - at BT 21CN rates it'd be north of £160. So if this problem came up on our BE line and affected our working day I think we'd have to look at moving ISPs or getting a second line and demoting the A&A line to a BT based service only used if the other line was showing problems. :(
That is no fault, that is simply "graph decoration" as a value-added service to you :D
ReplyDeleteAs the "lucky" owner of the line shown in the graphs above I can say state how annoying and inconvenient it is. The problem appeared quite suddenly on the 27th of January, and has continued ever since. As the graphs suggest, our connection is basically unusable between 1500 and 0100. The speeds we can achieve during that time are largely sub dial-up, and some web pages will time out rather than load. At the weekend, the situation is bad from about 1000 onwards.
ReplyDeleteTo say that I am disappointed with BE/O2s response to this is most certainly an understatement. They have provided a service which I was incredibly happy with, right up until there is a problem. For them to now try to lay the blame for that problem with "super users" is to ignore the issue in an effort to save money.
I am a very happy A&A customer, and will continue to be so. So far they have done everything I could ask, and been more than helpful with updates on the situation. However, if BE/O2 do not resolve matters quickly, I may be forced to move to a BT 21CN connection. Paying for an internet connection which is unusable for more than 70% of the time is not a sustainable option.
Thank you to A&A for all they are doing but, please BE/O2, take some responsibility!
Doesn't sound like your service is of merchantable quality. I do hope A&A haven't been charging you...
DeleteWell, a fault this long we would normally look at giving money back on the line. We want to find a resolution if we can, ideally Be/O2 fixing it, but failing that changing to BT.
DeleteApologies for not being clearer in my first post.
DeleteIn no way do I hold A&A accountable for this situation, nor am I asking for a refund/discount or anything else. I have always been, and continue to be, very happy with the level of service and support they provide. The best move I ever made was going with A&A, and I have no intention of changing or looking for recompense for something outwith their control.
I have also been happy with the service provided by BE/O2, right up until the end of January when this problem began. I would very much prefer that BE/O2 sort the problem out and give me back a decent connection again. Failing that, I may be forced to move to a BT connection until the problem is resolved. Despite all of that, I am not looking for a refund from BE/O2 either; just a reliable connection.
Problems happen, such is life. Taking responsibility for those problems and resolving them in a timely manner is how companies are judged. I'm really at a loss as to why A&A should be out of pocket for a problem which is not of their making, especially when they appear to be pushing quite hard to have it resolved.
Thanks for that James, and obviously we don't want you out of pocket either. We are still chasing with Fluidata now, and if we really get nowhere on this we will sort something else for you. Sorry it is taking so long. In the long run it will help us have a clearer policy to move much more quickly when other customers on Be/O2 lines get congestion. Fluidata were very well aware of our expectations right from the start when we first connected Be lines, which is one of the main reasons I am disappointed. Lets hope the take this on and get it sorted. If anything, making this more public could help Fluidata in their battle with O2 - I do hope so - it was not just to have a dig, honest!
DeleteWe did see one comment elsewhere suggesting this could be an issue in our network. I don't believe it was a serious comment, but just in case someone thought it was - this issue is in O2's network and they have confirmed that. The graphs are LCP echo loss/latency graphs from our LNS and allow us to identify where issues are very easily. The issue is only on specific Be lines in a particular geographic area - this is how we can be sure it is a backhaul issue and hence contact Fluidata in the first place.
ReplyDeleteBe/O2's resolution to this seems to be to send nasty-o-grams to their (unlimited service) retail customers threatening them with disconnection if they don't reduce their usage.
ReplyDeleteOne customer reported being told he used 243GB in the month triggering the warning. Not insignificant but hardly what you'd expect to cause critical issues.
Be/O2 give the impression of having no interest in upgrade while they complete their converged network.
A thought is that they've maxed out available backhaul ports they can plug GbE into and are waiting for converged network infrastructure to supply extra GbE ports as they either can't or don't want to deploy 10GbE.
Just to add to this I understood from a source that the Be network was largely Alcatel 7302 ISAMs, with backhaul going straight into an ISAM then daisy chained to any other ISAMs in the exchange as opposed to the approach Sky take which is to have Cisco 3750M switches for the MSANs and backhaul to connect to.
DeleteUsing an ISAM to terminate the backhaul means no 10GbE support, and 8GbE maximum which sounds fine until you think about daisy chains of exchanges and servicing any other ISAMs in the local exchange.
This could of course not be true, just what I was told. The Sky stuff I know is true, Be terminating backhaul straight into ISAMs rather than switches may not be. It would, however, definitely explain why the new network is required for these upgrades rather than ordering a new GbE from Openreach - a new switch in each exchange to deliver increased port density.
I recently migrated from BE retail (BE unlimited) to a 02 wholesale service, and during office hours there is a huge amount of jitter, which is visible on both f8lure and think broadband quality meter graphs, as well as an increased base ping to every target the only plus is that BBC I player works without issues unlike the retail side, And based on my experience i would no recommend anything Telefonica.o2
ReplyDeleteAs someone who has had both flavours BE retail & 02 Wholesale based of Telefonica's offerings
ReplyDeleteMy findings are that the 02 wholesale product was lacking higher ping times than the retail offering which i think is down to how they route from the exchange to the handover point , or due to some misconfiguration within the Telefonica network, I am also seeing a lot of jitter during office hours, this can be seen on the f8lure and think broadband graphs and looks like something is streaming but it is 100% not Both retail and wholesale are poor, how they can justify charging what they do i do not know
That is unusual as the usage pattern on most BE exchanges is low during the working day and high in the evenings.
ReplyDeleteCan you post a link to the f8lure graphs and what exchange are you on?
http://f8lure.mouselike.org/proxyfirebrick.asp?ID=50569 Exchange = LCSOU From 8-9am onwards each day,
ReplyDeleteI don't see anything similar any any other graphs - if anything it probably matches the usage pattern of the kind of customers on your reseller and their interconnect or network is busy.
ReplyDeletePossibly also shown in this graph from the same provider...
http://f8lure.mouselike.org/proxyfirebrick.asp?ID=50396
Although the line looks far more heavily used and unstable than yours but within the mess a similar pattern emerges in the historic graphs.
I don't see today's news helping improve things with BE at all ;-)
ReplyDelete