Outlook and IMAP: A Fatal Combination?

We’ve received reports of Microsoft Outlook users who use IMAP and appear close to hitting their storage quota receive an annoying and perhaps even debilitating number of pop-up alerts.

TL;DR: Raise your quota, clean out your mailbox and switch to a functional client. Or any of the 7 other options.

Let us set the scene:

  • A user receives 20 units of storage, and uses 19 of them — or 95%.
  • The IMAP server is configured to issue an ALERT response when the user exceeds a percentage of storage used, configured on the server. These warnings are issued once per connection created.
  • RFC 3501 dictates that the warning MUST be displayed to the user. Note that the RFC mentions drawing the attention of the user — Outlook implements a work-flow impeding alert window that needs to be clicked away.

When we get in to the nitty gritty details of the situation however, it turns out:

  • Microsoft Outlook creates many, many individual connections. A switch from one folder to another will create a new connection, as will sending a message (b/c saving off to the “Sent Messages” folder on IMAP).
  • Microsoft Outlook appears to adhere to the RFC, albeit in a fashion that creates a very poor user experience.

So, is it a problem of Kolab, or a problem of Microsoft Outlook, or both?

This is a tricky question. Nobody can fix Microsoft Outlook — and I mean literally nobody, because Microsoft must surely have tried.

This, however, does not imply it is therefore Kolab’s problem to resolve. It’s the client that renders the poor user experience while Kolab behaves as it is configured and expected to. Now, we may have some options in mitigation, but they do not amount to the resolution of the original problem; the user is close to using more than what is paid for, and dire consequences do not meet the expectations unless the user is warned prior to the consequences kicking in.

Option #1: Disabling quota warnings altogether

Disabling quota warnings will help the Microsoft Outlook user that uses IMAP, and is about to run out of storage, but eliminates the feature for all other users.

Additionally, disabling quota warnings to suppress warnings about a close-to-dire situation eliminates the need for the user to become aware of the need to either raise their quota or clean up their mailbox.

Please note that the effect of running out of quota altogether is to imply that new emails sent to your account are bounced. The original sender will receive what is called a Non-Delivery Report (NDR).

Option #2: Have the user switch to using ActiveSync

Outlook can also connect to Kolab Now over ActiveSync — which would then include calendars and address books synchronizing without the need for a plugin or a separate account configuration.

This is an interesting proposition, because ActiveSync does not appear to provide a mechanism to issue these kinds of warnings to the user. In effect, this makes it Option #1 but for Outlook only.

However, our current implementation of ActiveSync flattens folder hierarchies for calendars and address books. This is a residual artefact that remains from the days that most ActiveSync client implementations (Android and Outlook, to name two) did not support multiple folders. As a result, the “multi-folder” capability was defaulted to “no”, and devices that did support it needed to be white-listed explicitly.

Today, I would invert this logic — most client implementations do support it, or should support it, and for the few legacy client implementations that are still around, or even those still lagging behind, of which there should ultimately be fewer and fewer, we can create the black-list for the inverse of the new default.

We need to understand however, that this seemingly very light-weight, superficial, easy to comprehend change has severely complex implications when applied to environments with tens of thousands of different devices already registered and already synchronized. In other words, this won’t happen overnight.

Option #3: Filtering quota warnings

A somewhat intelligent IMAP reverse proxy could, in theory, apply a filter to suppress warnings and remind the user once-or-twice-or-trice a day or so.

We have such an intelligent proxy (called Guam), but it does not have this rule for it. I suppose we could go through the inception and elaboration, but the tracking and sharing of state across connections is not what it has been built to do. In other words, neither is this going to happen overnight.

Option #4: Tweaking the server settings

We can tweak the server system settings to take in to account a chunk of storage such that the quota warnings are issued only when the issue is equally urgent to the degraded user experience in using Outlook. However, if 20 units of quota compare to 20 MB of storage, we need to understand that currently using 19 MB is going to need to issue a very, very visible and annoying warning somehow. However, if the 20 units amount to 20 GB, surely a chunk of 1 GB of free space should allow the user some leeway in getting all annoyed about a very badly and poorly implemented client application.

I would specifically refer to a setting called quotawarnkb, for which I will run though my proverbial circles and proof the change I recommend to my engineers equates to a fairly balanced suppression of warnings yet does not make mails bounce without prior warning — which, when ignored, are considered consent to the system applying the consequences.

Option #5: Raise the user’s quota

When a user needs $x storage, and is configured to be allowed $x storage, and then uses $x storage, yet the user needs all that is contained within that $x storage, the user needs to consider +$y’ing that $x storage.

The user cannot stand and say, “I paid for $x, but the client I use is so poorly implemented, I therefore require a free +$y to shut up my client”. We can consider Option #8 as the courteous approach, but we cannot endlessly continue to apply margins on top of one another in order to prevent a user from experiencing the full extent of (un)friendliness of the client application they choose to use.

The user can, however, pay for $y — instantly relieving the user of dealing with the symptoms of the implementation, without immediately requiring a clean-up. It’s 50 cents per GB per month, and should result in a negligible increase in the overall cost, of the other choice is to. It’s the kind of pricing that allows you to not even have noticed I didn’t mention the currency — until just now when I pointed it out.

The user can even raise the quota temporarily, for he or she may have no time to clean up today or even this week. Should the user decide to have the quota raised temporarily though, we would still get to Option #6 eventually;

Option #6: Clean up the mailbox

This turns out to be what it comes down to most of the time — no extra cost may be incurred, but the problem must be resolved.

Even when we resolve today’s occurrence of the problem (see the other options listed), the user’s consumption of storage is set on a trajectory that triggers the annoying client implementation.

Option #7: Defer to Microsoft customer support

If the client application is bought and paid for, and the license is considered valid, you should have no issue calling the vendor out on the delivery of a product that does not meet your expectations. Not unlike what you’ve been doing to Kolab Now support.

Option #8: Shift dealing with the margin to the provider

This option would include functionality that translates your 20 units of storage paid for to just over 22 units of quota. Now, should you hit the 90% threshold in consumption, you’re consuming right at the boundary of what you paid for, and the debilitating treatment of the warning is arguably better justified.

This must still be understood as working around a client misbehaving, though, and won’t solve the usability problem for when the user does hit that 90%.

Option #9: Pay as you go

We could aim to remove quotas altogether, and transfer to a pay-as-you-go model — the more you use, the more you pay. The less you use, the less you pay. No quota involved.

This would be a big change, and would need to continue to work with actual quota (for business needs).

Option #10: Switch to a client that works

Thunderbird, Evolution, Roundcube, Kontact are all well within the reach of users, but some will insist on using Microsoft Outlook instead. Some poor folks may even be required to. However (in)valid the reasons, there will clearly always be Outlook users to our service.

Summary

So here’s what we’re going to do;

  1. Establish a minimum amount of free storage (in kilobytes) that delays the quota warnings until they become actually problematic.
  2. Document known solutions to getting a dysfunctional client back to being functional again (i.e. raise quota, clean out mailbox, etc.).
  3. Work to enable to make the threshold of 90% consumption equate to the 100% boundary of units paid for.
  4. Switch over the logic in ActiveSync that flattens hierarchies by default and is provided a whitelist, to blacklisting devices for which the hierarchies need to be flattened.

 

 

Advertisements