Slartibartfast tour!

I have a couple of very long standing (aka suffering) mates, and we did a thing last year which we hoped to be the start of an annual thing. We went to LA and Vegas.

Mike organised it and we had 1st class flights to US. That was pretty cool. It made the flight pretty bearable, but still for like 9 hours, was hugely expensive. It was good though. We had fun in LA, and even introduced a barmaid to marmite! We had fun in Vegas. I went hunting a new camera, nightmare, and I probably caused annoyance to my mates in doing so. I have it now - Canon, please don't do a new camera in next three weeks.

It was an awesome holiday, and I was really appreciative that I was not doing the driving (LA to Vegas) to be honest. I also appreciate how much Mike paid, but the whole trip was great fun.

This year, I am carrying on the tradition, and have us booked in a couple of weeks on an 8 day cruise. We were lucky, Mike found the trip after we all tried the web site(s) repeatedly. We assume a cancellation to get this cabin for this price on two weeks lead time.

This is not cheap, and neither was last year. It is each of us saving up to afford this, and then only just. It is all about making something memorable and worth the money. You only live once, and it is really nice if we can afford to live, occasionally. This year a series of favourable events mean I can manage to pay. I suspect Simon is bricking himself for next year, but to be fair it is not about the price but the experience and being with friends for a week or so, and so no pressure :-)

The "cabin" (aka "stateroom") we have booked is fucking awesome, sorry. Hot tub, piano, ten times the size of all but two other cabins on the ship (one of which is the other one like this, there are only two). You could throw parties in this. I'll get pictures and videos. I could not believe our luck when it came up. The web site then made it impossible to book it and I spent well over an hour in Thomas Cook today while the agent there (that we know quite well now) sorted it all. Well done.

It is the Slartibartfast tour - and yes, spell checking understood Slartibartfast! Norwegian fjords. With any luck some northern lights. We'll all be working to some extent - that is the nature of each of us running a business, but the open bar and wifi will help with that. I may be working from the exclusive hot tub with open bar, sorry...

Really I can't wait. It is going to be so cool. My wife is jealous and I just know I'll be paying for that somehow.

"I don't want porn coming in to my home"

As you know, I always get very concerned if any customer has a disconnect with what they expect and what we provide. I take it very seriously and always try to improve how we work to avoid it in future. It does not happen often, but we had one today that was, for want of a different word, "special". I initially assumed it was a wind up even!

The gist of it was that there were a list of things that she did not want coming in to her home including porn, suicide, self harm... She was amazed that "the internet" has not taken down such videos. Who? Apparently "blue wale challenge" is real and "every school in the uk has sent a formal email about it to parents and children have been told about it in assemblies" - really?!? Not the schools round here, sorry.

OK, personally, I think that if her kid's school has told kids in assembly not to google for suicide videos, that would be something to complain about as it would be very irresponsible. Tell a kid not to do something, that works every time, duh!

Though, her kids are apparently quite savvy, as her son "innocently googled some games the other day and found a pretty hard core sex games internet site"... Err, OK... Safe search, anyone? Given a later comment I hope this is a son in late teens... If so, I suspect he knows how to "innocently" google many things by now. Time to talk to your kids about what you may find on the internet I think.

OK, lets be a bit fair here. Not everyone knows how to fact check stories. People do need some help understanding how they can filter content, or even just turn on google safe search on the browsers their kids are using! It is actually quite a concern that parents get very little help in this area - it is bad enough learning about everything you need to know when being a parent but for a whole generation, this stuff is new and complicated. It is not something parents could have been taught in school even.

To be clear: we are more than happy to offer advice, and even set alternative DNS servers as default on the router. We're not irresponsible here. What we do is make sure every customer is well aware that they are buying an unfiltered service with an active choice you cannot ignore on the order form, as well as confirmation in the key terms you have to tick, and on our web site and the order confirmation and the information pack we send.

It is also very important that parents understand that no filtering is 100%, so is a "false sense of security" to some extent. It is also the case that a teenage boy (and girl I expect) will be more than capable of bypassing filters if they want to access something. What you need is education and openness, not cotton wool. Else you create innocent blobs that go out on their own in to the wild world of the internet at 18 and don't know how to handle what they find and probably with nobody to talk to.

But the icing on the cake was the last bit...  "My boys play 18 rated pc games...... all killing and horrrible stuff......Hopefully they will still be able to play these games?"

OK, now I am not sure what to say, sorry... Please, just be a parent!

The plot thickens...

Unbelievable...

Today I decided to try something else with the Apple TV. I set up another Apple ID with iTunes store account. I then family shared my normal Apple ID to that.

I logged in as the new Apple ID and played episodes via the family sharing.

First episode fine.

Next episode - wants my iTunes password before playing...

And before someone asks, I also tried fixed IP config IPv4 only already.

Next step: physically swapping for another Apple TV

Update: didn't help - Apple TV using my account asked for iTunes password on playing one of the episodes. Arrrrg!

Cursed apple

So where are we with my Apple TV saga...

They are referring to "engineering". That is it...

Latest things I tried...

• Change apple ID, so if any machines I forgot are trying to log in then they will not know new apple ID.
• Only have my Mac, Phone and Apple TV on the new Apple ID
• Turn 2FA back on
• Turn off home sharing, just in case
• Log off iCloud, just in case
• Again, confirm payment details, and actually buy a film on the Apple TV

Guess what - not helping. Still asking my iTunes password, around every hour, on play of episode, or rewind within it. Randomly will not even show episode with "unexpected error" requiring restart.

This is what makes pirating movies easier! If I pirated movies I would not have this hassle.

I expected better

I had an interesting exchange with a customer this evening.

His internet has been down for an hour and he texted our "Major Service Outage" number saying so and saying he "expected better".

I'm not sure what to say to that - if we really have a customer that expects that a fault, which they have not even reported, is fixed within an hour, out of office hours... What does that mean? If we really have people with that expectation I am tempted to try and sell to them. I expect it will be many thousands of pounds a month for such a service, with multiple redundant leased lines, diversely routed, and a lot of 24/7 active monitoring and staff. I suspect we could do it, for the right price.

Of course, that is not what we normally sell. So if one of our customers expects that they have a serious misunderstanding as to what they have purchased. I try to be honest in what we sell, so I am worried that someone expected more.

At the end of the day things calmed down. I was unhappy as he "abused" our MSO system. It alerts many staff for what could be a major outage. In the evening, not so bad, but middle of night, not good at all. Either way it is staff that are not at work and trying to spend time with family. The reaction from staff that do get the alert was not good, and I took over and was, I have to say, rude...

As per the web pages on this, I was rude. You are asking for a rude reply if you abuse the MSO system! That is what we say!

It turns out he read of the MSO text details on the status pages, which does not have as many caveats. We need to fix that. It is clear it has to be "multiple lines" and so was clearly abusing the system. It was not as clear as the main web page on this. So we can be clearer, I conceded. We'll fix that. Sorry.

Ironically, this fault looks like it may affect 3 lines. It is even an "incident" that TT have spotted. So ironically not something we need to take any action on as it is all in hand. The issue is that this person did not know it was impacting more lines. Our aggregate systems do not spot three lines going down together as that is too few to recognise as a pattern. Good news is it should be fixed soon.

I hope the mix of rude and reconciliatory messages was the right level in the end. We'll see. At the end of the day, expecting a line issue to be fixed within an hour of *not* reporting a fault is special, in my opinion - and I'd be happy to sell such services for the price they deserve...

Let's hope this line is sorted soon.

Talking to apple support

I went through the iTunes email based support, took ages, and no help.

So I called 08001076285, as they suggested, to get support. It is a horrid voice based IVR thing. So I have to say my serial number the first time. This time I said my case number.

Oddly, they are actually very pleasant, so well done on that. I have a dread of calling any tech support normally. I hope our (A&A) tech support does not create the same sense of dread for customers. We have no call gate!

On the first call I explained the issue, and what I had tried. Eventually they suggested one thing I had not ("restoring" via iTunes and a cable). So I have done that now, and hence another call.

I am tempted to publish the call recordings (and annoying hold music). I will have to change my security questions first :-)

Basically, I have tried everything so far...

• Changing password
• Turning on two factor authentication
• Turning off two factor authentication (twice now)
• Removing all devices from account and starting again
• Turning off family sharing
• Factory resetting (many times)
• Restarting using iPhone, and using manual setup
• Using cable to "restore" the Apple TV via iTunes
• Upgrading, many times

Still broken.

They think "something somewhere is asking for an iCloud password, or something" and that is upsetting it.

I removed all devices from my apple ID already, if this is what it is that is mad. I have just logged in to two iMacs and set new passwords. If that solves it Apple are fucking stupid! No way a third party failed login attempt should break my Apple TV. Just broken logic there Apple!!!

P.S. If this works I should do a more controlled test - what if someone just used my apple ID and wrong password, would that break it? If so Apple really are broken!

Julian Huppert - Cambridge

I have met a few MPs, and Julian is (was) one of them (see wikipedia).

The party he stands for is the Lib Dems, and the area is Cambridge.

I have met him, and discussed several policy issues with him. Overall he is sensible in my opinion. More so than many MPs and way more so than some MPs or the PM.

He is educated, understands science, and even rides a bike!

I was not really in to politics until some of the more recent stupidity that started to affect me and my customers. So far Julian has been quite sane in the discussions on the many issues that have come up, and so have the party for which he stands.

Now, some would say this sounds like a mediocre endorsement - but please, those that know me -  this is far from mediocre - I do not endorse someone lightly.

Julian is someone with which I feel that I could have a sane debate, and would listen to my views. If I lived in Cambridge, regardless of the party for which he chose to stand, that would be good enough. He should represent his constituents, and I am confident he can do that. He has integrity and sincerity.

Right now, in politics, we have some crazy shit going down, and the Lib Dems actually seem to be the only ones being vaguely sane right now.

Even if you are not massively in to one party or the other, if you live in Cambridge, please do look at the candidates and take Julian seriously. If nothing else, he is one of the few who could be in parliament with some decent education and understanding of science.

So, if you can, vote for Julian in Cambridge. A voice of reason in these troubling times.

PGP usage

PGP (Pretty Good Privacy) has been around for quite a while now, and the GNU code for it (GNU Privacy Guard, or GPG) is free.

Amongst other things you can encrypt and/or sign emails using PGP.

Unfortunately it has yet to catch on for a common usage. We use it a lot in A&A. We sign the emails we send in almost all cases and have done for decades (I like how I can say "decades" now when referring to A&A). We are just starting more comprehensive encrypting of emails we send as per another blog post.

But it is still uncommon. It is not properly supported in almost any common email clients. I use thunderbird and there is a good plugin (enigmail) which works well, but still a plug-in. It puzzles me a bit as to why it is not a lot more standard in major email clients yet, after all this time.

Of course, one of the big problems, is the "trust" of keys. There is (deliberately) no central authority. Sadly, a central authority model, like that used for https, is way easier for end users. They could automatically trust an email claiming to be signed by their bank because it would be signed by a chain of authority their email client knows to trust. This is the same as the way you can go to https for you bank and know it is them.

There is an email system for this, S/MIME, but even support for that is complicated and not simply included in major email clients, as far as I can see. It also has the problem that individuals want keys, and a central authority model makes that a pain and probably involves paying to have your key signed.

I do think some organisations could do more to encourage PGP. It would be great if Companies House, for example, would sign company keys as a service that is part of managing company registrations. They already have security measures, and they could use the fact they can trust a signed company email as an added feature in dealing with companies. That may encourage more companies to check signatures, and maybe even use company key signed emails as signatures for contracts.

You still have the issue with individuals, but again, organisations that already do security checks, like banks, could easily include key signing. It would be a way to advertise their bank as a source of trust.

Anyway, enough of solving the problems of the world for a moment, the main reason for writing this is that I have to assume more people are using PGP at last...

How do I know? SPAMMERS!!!

I am seeing more and more spam that includes a PGP signature block or a PGP public key block in the spam email. These are usually broken or bogus, which is silly, but they almost certainly look close enough to get an improved anti-spam checking score, and give an impression of more credibility to people.

That would only happen if people really are using PGP more. So, interesting times.

Of course, if people do use PGP more, then spam checking can start actually checking signatures and trust chains, as part of the scoring. Get enough people using PGP, even if only for signing, and we could ultimately eliminate spam (ha!).

Serious rants at apple now

Progress on iPhone roaming

For whatever reason, the instances of the roaming issue have massively reduced in my house. The main difference was that all APs on same PoE switch, but could be the phase of the moon for all I know at this stage. It is a bugger to track down this one.

This means it is taking days to "catch it in the act". The good news is that this happen last week, and I confirmed there was good signal but no connectivity - no IP or anything even to a device on same AP. So I changed config to be fixed IP.

Today it has happened again and we have learned some concrete details of the problem. Also, it has happened in my study, and so I have the phone in the state, captured, and on charge, sat here. It is not between two APs, so should stay broken.

So what have we learned so far?

The phone was set completely static IPv4 config, so no DHCP. This means the problem is not trigged by the way DHCP works or by the FireBrick or gateway doing DHCP in an odd way - that eliminates a load of possible concerns from previous testing. The fact that many people came forward with the same issue on non FireBricks was also a relief.

The controller for the APs claims the phone is not attached, it shows it was, but that it is not now. This is a clue. The phone thinks it is, and shows full signal. So the underlying issue here is a mismatch so the phone thinks it is associated and the APs think not. This has to be a big step forward and suggests it is the roaming process itself failing somehow.

In this state (perhaps unsurprisingly), even with the fixed config, we cannot get any packets to flow, even to another devices on the same AP (and subnet).

What next?

At this point, I am keeping the phone on charge in here in the broken state as long as possible, and have set up firewall access for Ubiquti engineers to have full access the APs and the controller and see what they can find. I hope they find more clues to the problem, but I appreciate it is tricky with some issues like this.

We're doing all we can to get to the bottom of this.

Update...

The phone was in the same state having left it all night. So I started to do monitor-mode wifi dumps on my MacBook as requested (wireshark is working quite well on MacOS now). On the AP in here I did not see the MAC of the iPhone at all. I've sent them the dump anyway.

Sadly, trying to get laptop on another channel to dump that I made a config change to APs, which made the phone spring in to life... That has to be a clue for them I suspect.

So...

• Not DHCP related
• Failure mode is phone things associated and AP thinks not
• We know wifi off/on on phone fixes
• We know roam to another AP on phone fixed
• We now know reconfigured of AP (even leaving SSID in place) fixes it

Ubiquiti think that any packet from the phone which thinks it is associated should cause a de-auth from the AP which should cause the phone to re-connect. They can't dump that on the AP, hence monitor mode. Sadly I did not capture any packets from the phone on that channel so not conclusive.

Apple TV has gone terribly wrong for me

First off, my Apple TV appeared not to work - I swapped cables, nothing! So I got a new one, same model. Worked! Yay!

Simple enough, though sadly since then the new one did the same a few days later and I now conclude it may be the port on the TV that is being flaky, so on a new port - bugger. I may have chucked out a working Apple TV. Idiot!

However, that aside, the new apple TV is being odd.

Yes, I signed in to iTunes. I have 2FA set up, but that is not actually the root of this issue as I tested today. That all worked, all my stuff shows.

Problem is that every new episode of something I want to watch, even though already purchased, is asking for my Apple ID password, every fucking time.

To start with it popped up on my iPhone as well, so I had a keypad to enter it, but even that has now stopped.

It is making watching anything on Apple TV unbearable and I have no idea why. I have already removed the old Apple TV off my account. I tried removing 2FA (and have since put back). The best I have now is dictating my password for EVERY FUCKING EPISODE!!!

I have no idea why - I even told the Apple TV not to ask for password for purchases, yet it does, many times, and the still asks every time you try to watch every episode.

I will have to ditch it if this keeps up.



P.S. tried all the suggestions, thank you all for them. Finally removed from iTunes, factory reset, upgraded, and re-set-up. Seems to be being sane now again. But really, what a messed up failure mode.

P.P.S. Bollocks, that was fine for two episodes and then again is asking.

Customer Privacy

We have completed the first step in providing customers with extra privacy by encrypting emails to customers if they wish.

This is currently only the accounts system. We are extending it step by step to other systems.

I spent some time working out the best way and I think I have something sensible. On the accounts system, once logged in, you can update contact details by a link on the main page.

This allows contact detail to be updated, including email address, but also allows you to paste in a PGP public key to use.

We don't care what UIDs, emails, or trust there is set in that, as long as not expired or revoked, we will use it to encrypted accounts email to you.

We then email you using new contact details (including encryption key) to confirm, and you have to follow a link. That proves you control the email address and the secret key. Once done the account is updated to use those details from then on. We also email the old details (email and encryption settings) to advise of the change just in case it was not really you!

RevK, thanks, first ISP to use PGP for communication with me (and i work for one) :)

We also allow some controls of emailed content, so text email plus optional PDF and optional XML. You can select PGP/MIME or not (i.e. just signing and encrypting the main body). You can even select if we include a confirmation link in the email or not.

The next step is to cover two main areas - call recordings and KCIs. KCI is Keeping Customer Informed and relates to all the texts/tweets/emails from the control pages. It will take some time to get everything on the control pages moved to KCI.

The principle is likely to the the same - load a key and we will use it.

I think this is an important step for privacy for customers.

P.S. We have had options for a long time on what is emailed, e.g. no itemised bill and no link to get it embedded in the email. This is extra protection to protect the entire contents of the emails. We may add extra layers to protect subjects in due course.

Gravity plating

In sci-fi there is artificial gravity.

But it occurred to me gravity is more complex that just saying the "gravity plating" on your Starship Enterprise is set to 1g.

1g, or 9.80665ms² is the nominal average on Earth. It is a result of being 6371km (ish) from a centre of gravity of a mass of 5.972 × 10²⁴ kg.

Basically the force is based on a constant times the mass of each object divided by the distance squared. So the force per mass of the object on the surface (i.e. you) is based on this constant times the mass of the earth divided by the radius squared.

But you get the same end result if the mass of the planet on which you are standing is much higher and the radius much lower. If you stood on a super dence object only 1km wide you could experience 1g.

There is a difference though, now your (near) 2m height means that the radius is noticeably different when considering the gravitational force at your feet and your head. You would feel the difference I expect! If you jumped in the air you would quickly have much lower gravitational force - reaching escape velocity would be way easier, surely?

Take it to the extreme, a 1m radius object with enough mass to be 1g at your feet, what is that at 3m from centre of gravity, 1/9th g? Light headed or what - jump and you are gone!

So the gravity plating on starships is not just a matter of being 1g, it is also about the apparent distance involved, surely. But the gravity is not exerted much beyond the ship, if at all. Lots of episodes show this, so it must be the like 1m radius Earth... Emulating a super dense but small structure.

Do all star trek people get used to being so light headed?

If we lived in such a world, would we evolve to be midgets?

These are questions we need to ask, people...

P.S. As per one of the comments, creating gravity by spinning part of the ship has the same issue. You can have a small radius spinning fast or a large radius spinning more slowly. The effect will be 1g at your feet but the change in gravity at your head would depend on the radius. I really had not though that gravity at a point in space has both a force and a rate of change like that, but it is obvious when you think about it! It also means that gravity because you are simply in a box that is accelerating is different yet again, indeed, you should be able to detect that this is not the same as gravity on earth, from inside a sealed box, even if you measure 1g, because it is not different at different heights in the box.

Barclays on-line banking bug

I reported a bug to Barclays months ago and they just ignored me. I have now complained about the time taken to respond to a complaint, and they are looking.

The bug is pretty simple, and very stupid.

The business accounts summary page shows "Last night's balance"

But it is not in fact last night's balance, it is in fact the current cleared balance. The actual last night's balance is shown when you go in to the account.

It is a stupid bug, and should be very easy for them to find and fix.

What got me was , now they are looking in to it, is that they have asked me:-

1. Computer details - e.g. Windows/Mac and what version of OS
2. Browser details, and what version
3. ISP used

Now, this is a web page, via https. Why would the computer or browser matter. At a stretch, they may format the content differently for different browsers or operating systems, perhaps. Seems unlikely to me, but that could be the case, just maybe. So OK, ask that I suppose.

But why ask about the ISP?

Surely they do not think that the ISP can, in any way, influence the content of a secure web site? If they do, then why do they trust the use of secure web sites in the first place. Why ask the question? I have asked why and not told them as I suspect it would confuse them.

Next step in AP testing here

I have tried quite hard to get the three APs here to break when using a FireBrick FB2700 as gateway on a separate subnet (i.e. WAN side of FB2700 on my main LAN here).

What we did is move from a set-up that broke on my main LAN, to a separate subnet off the main LAN and a Ubquiti EdgeRouter. That worked! So I tried an FB2700 instead in same set up, and that worked too. So it was splitting off to a separate subnet with some sort of gateway that seemed to fix this somehow (rather than specific choice of gateway equipment).

My working theory was that there must be some network set-up aspect that is somehow triggering this issue (whether that set-up is a bug or error or not). This would account for why FireBricks seem to be a common factor as well as Unifi and Apple. FireBricks are not an off the shelf linux system so have very different default settings, and maybe that leads to the problem set-up to be much more common. Well, it was an idea.

Ubquiti had the problem immediately with an FB2700 that we sent them, so sounds like a default setup with very few changes would trigger it, but it did not do so here. I have now gone through matching settings to the gateway on my main LAN. This includes things like leaving DNS to automatic which announces the FireBrick itself as one DNS server only on each of IPv4 and IPv6. I even set up the extra VLAN for guest WiFi which is separately firewalled but on the same subnet with proxy ARP/ND between the two LANs, just in case that was a trigger somehow. After some days of doing this now, it really is "just working", which is rather frustrating.

So this morning I am back on the main LAN as before. Hopefully this will "break" things once again and hopefully quite quickly. It may be a few days to be sure.

The techies at Ubquiti have advised that a pcap on the actual AP itself may help, so the plan is, when it breaks, leave my phone in the broken state (don't move it) and try and diagnose with pcaps on the APs.

To further diagnose I also plan to set the iPhone with static IPv4 config, as some sort of "DHCP throttling" may supposedly be to blame for this. I have double checked with the other developer on FireBrick, as we have both worked on the DHCP server, and neither of us know of this "feature". However, it is worth investigating every avenue. Previous tests (albeit years ago I expect) showed the issue still happened with no DHCP involved. The problem may have changed since, so I'll repeat those tests to confirm. I'm not going to dismiss any ideas.

In case it is not obvious, when this started, years ago, the first assumption we had is that it has to be the FireBrick at fault, and I spent a long time testing things like static config to eliminate DHCP, and checking packet dumps very carefully for DHCP, ARP, ND, RA, RS protocols to try and find anything that would point to FireBrick as the cause. Only after all of that testing did we raise with Ubiquti.

I'll keep you posted...

P.S. Finally (Thursday) my phone failed, I confirmed even a static config could not send or receive packets, even to a device on same AP. I confirmed roaming to another AP does fix. I am leaving on static IPv4 config now to test.

Working with ubiquiti

This is a separate post as something seems to have kicked off on twitter this morning. And first off I'd like to apologise to Brandon from Ubiquiti for swearing.

Ubiquiti have been very helpful trying to get to the cause of a long standing issue impacting a small number of people, but including myself. It is a very frustrating issue which has led me to consider scrapping using the Unifi APs on more than one occasion, but I do like the Unifi kit and I would like to get this actually resolved and continue selling it.

What do we think we know?

• This only seems to impact Apple - it is seen on iPhones mostly - not android.
• This only seems to impact Unifi APs - not seen using other APs yet.
• This almost always seems to be FireBrick as gateway router (at least one case of not FireBrick)
• This is a rare situation, with many people using hundreds of Unifi APs with no problem. Similarly lots of people using Apple with no problem. Similarly lots of people using FireBricks with no problem.
• It seems sticky - when a set up has the issue, it stays. When a set up does not have the issue, that stays OK. It is also very intermittent and can seem to take days to be sure if fixed or not.
• This seems to be only where IPv6 is on the network, which is one reason most people don't see it, and may also be a reason why cases where an IPv6 friendly router sold by an IPv6 friendly ISP is the most common case we have seen (i.e. why FireBricks in almost all cases).

As I say, Ubquiti have been very helpful - they sent us two switches, and edge router and a security gateway. I was only expecting a switch from what was said, so thank you. It has allowed more testing. We sent an FB2700, which has also allowed more testing. The results are interesting, to say the least.

• Brandon has advised that using FB2700 they see the problem right away. This is good, we have created a set up with the problem. He confirms that using other gateways he does no see it. So something about the network when using a FireBrick seems to be able to trigger this somehow. Oddly he has also seen up to 60 seconds "delay getting an IP" which is not one we have seen. The problem we have seen is permanent - you lose all IPv4 and IPv6 on a roam (intermittently) and do not get an IP even after 60 seconds, all you see is the 169.254 address for when you don't get a reply. I assume that is not what Brandon was seeing, but actually a "delay", which is rather odd. If it is, then that explains the phantom delay and means he has exactly reproduced the problem.
• Here, we tried moving all APs on to a unifi switch connected to our main LAN (and using FB6000 as gateway). It did not help. That eliminates the switches I have which could have been messing with multicast or something.
• So I set up a separate subnet for the APs, connected to a Unifi switch, and that then connected via their EdgeRouter. Sadly I needed help setting up IPv6, but got there, in spite of some of my typos. It seemed to fix things - great.
• So I changed to using an FB2700 on the same separate subnet and same Unifi switch, just swapping one box, and again it is working. I have made the set up as close to the main LAN as I can, same VLANs etc, and the APs are the same config exactly - not changed.

This means the separate subnet appears to be the fix rather than change of router.

It also means a really simple set up of FB2700, switch, and three APs here just worked, but Brandon, with presumably a similarly simple set up, immediately failed. That would be nice to try and compare.

The roaming also seems to happen, apparently as expected, with no interaction with the gateway. No DHCP or anything, just switches over from one AP to another. So it is hard to see how any gateway can be the cause of the problem.

At this point I am wondering if somehow it is a specific configuration of a network that breaks it - I hesitate to suggest the actual IPs in use somehow. I also wonder if it is something else on the LAN causing this - but that does not fit with Brandon's comments.

Unfortunately we have reached an impasse with Ubquiti - they have been very helpful up until now, and thanks for that. But even though this only happens with their APs, and only happens with Apple products, they have now concluded it must be FireBrick and "So at this point I don't think it's fair for you to ask us to help you resolve this.  In doing so your are asking us to help your company make a competing product, for free." and now "So I'm out. Refuse to interact under such disrespectful terms."

We'll continue to look for the issue. I suspect, when we find it, it will not be something where any finger of blame can be pointed at a single bit of kit. But nice to know the spirit of co-operation is alive and well, up to a point. Thanks for your help so far.

FYI, I don't care that Ubuiti have a "competing product". As an ISP we work with competition all of the time for the greater good. I'd be happy to continue to work together to get to the bottom of this anyway - all of our customers would benefit from that. I will, of course, share our findings, even if we find a bug in something FireBrick is doing.

P.S. My next avenue of investigation is differences in configuration, no matter how small, to try and see if we can find a network set-up difference. It is very likely that a typical (mostly default) FireBrick network will have some notable differences to a typical (mostly default) non FireBrick set-up...

P.P.S. You gotta love it - Brandon has complained to FireBrick about one of their employees (me) swearing at him. This is from the country that actually believes in free speech.

AAISParty - 20 years old

Victoria did well co-ordinating it all. Pictures (here).

We are obviously interested in feedback - we do not do this sort of thing often.

There are a lot of things to arrange with a party...

• Food: We had a professional BBQ laid on, but also sweets which went down well (more with the adults than the children I think) and at the end of the evening pizza. We have some sweets left over, but mostly I think we had the right balance and nobody was left hungry.
• Music: Some music to start then a live band on until the end. What I liked is we had the marquee which was louder and the live band, but the meeting room in unit 2 close enough to hear the music but quiet enough to talk to people. A good set up.
• Drink: A range from 7up, water, fizzy water, diet coke, coke, fanta, J2O, alcohol free Becks (nobody wanted), Becks, Kopperburg, more cider later in evening, whisky (nobody dared open, sorry), bottled ales, coffee, tea - I think we had everyone covered one way or another.
• Geek: We had a special event amateur radio call sign GB2OAA set up, and (supervised) use of radios. We had a slide show showing staff and a history of 20 years of A&A which was surprisingly popular. We had WiFi. We had detailed Q&A on how the network works (by me). We had our museum of old routers and phones (and "the internet") as well as showing people bare board FB2900 PCBs. We had a DALEK and Orc!
• Cuddly toy: Many FireBrick dragons, and the kids that came loved it all.
• Weather: As ordered, sunny and warm and dry, perfect... Thanks for that deity of your choice.

It was nice to meet some customers.

What could be improved - well the name badges went down well and we managed almost everyone, but a key lesson is not leaving it to eventbrite. We need irc handles and maybe twitter handles on badges next time as well as WiFi details, etc. But they are a souvenir for those that came.

On a personal note - I was worried because of Thomas. We had him on one of the slides even (thanks to my daughter for allowing that). One person said they had not spoken to me since this happened and asked if I was all right. Even with my daughter there, and her partner, nobody made any fuss. Thank you all for that. I was really actually quite worried. Nobody wants to cry in front of a tent full of customers!

Encrypting emails to customers

We are making progress...

The new email library is working, and the accounts systems was updated last week. This was not without some snags. On Friday we had some people unable to open PDF attachments. This turned out to be a quirk with an old exim on the server, which is being replaced soon, and created additional unexpected whitespace in the base64 coding of the PDF. From what I understand whitespace is valid anywhere in base64, so I don't think we were wrong, which explains why the tests with various email clients before hand (Thunderbird, Apple, etc) were fine. It was failing on some Microsoft mail clients though. We fixed that on Friday before main billing run Saturday.

A typo in the fix, or rather after the fix was tested, meant no invoices Saturday, which has been fixed today and all April invoices have now been emailed. One person with an old mutt script broke but otherwise all looking good.

The next step is part of the development work on the KCI system. (KCI is Keep Customer Informed). It will be used for status updates from the control systems tracking orders and faults and appointments. It is not used for invoicing or call recording. However, a key part of the KCI system involves customers registering public keys with us.

Customers will be able to email a public key to us with their email address on one or more of the UIDs. Once done, they will then be able to go to priceless (accounts), or clueless (control pages), and select from one or more public keys we have seen with that email address. That will select the specific key we should use for encrypting email to them.

This should avoid issues with rogue keys that happen to be created with the same email addresses. The customer, once logged in, with any 2FA and so on, will select the specific key to use.

This is important for privacy as invoices can include itemised bills, and call recordings could have sensitive information. The plan is all systems, including the new KCI system, will start using this key management to send encrypted emails, and our existing systems for ordering and faults will move to the new KCI system.

This may take months to complete fully, but should allow a lot more security and privacy for customers.