Egress is evil

How We Conquered the Challenge of Cloud Egress Charges to the Delight of Our Customers

“It’ll never work.” 

That was the comment from a brilliant consultant sitting across the table as we pitched our idea to a venture capital investor. A really, scary-smart dude whose job was to vet technology for a respected VC just told us we were doomed to fail. 

“First of all, I think the latency will be too great for challenging workloads. End-users won’t tolerate it. But even if you could somehow make it work from that perspective, the unit economics don’t work. The egress cost will kill you".

That last sentence sent a chill down my spine because it was something that we were already worried about. 

Egress is the notion of downloading data from the cloud to the edge (or to another cloud, region, data center, etc.). Some companies refer to it as “bandwidth” but however we label it, the concept is a bane to cloud users. Why is it so bad? Because it is expensive and unpredictable. 

It has always been our position that the concept of egress is largely an artificial construct, put in place to keep your data captured by the cloud provider. It costs you $0.025/GB per month to store your data but $0.09/GB to download that data? Come on. I’ll admit that there is a cost to bandwidth, but it is not $0.09/GB…

Cloud providers generally would like to think of storage as the basic infrastructure for higher-level services they want to sell. By having an expensive paywall for you to access your own data, you might be more likely to utilize those services. It has the added benefit of making it extremely expensive to pull all of your data out of their cloud should you decide to change vendors.

I’ve mellowed in my view over the years. I now give the cloud providers the benefit of the doubt, believing it may be more about guardrails than ransom. That is, they don’t want customers abusing their cloud by doing something like hosting a video service where they’ve got a small amount of data that gets streamed or downloaded hundreds or thousands of times. 

However, even with a charitable lens regarding the reason for egress, we can’t deny that it is the fly in the ointment. Not only is it expensive, but customers can’t predict what the cost will be until after the fact when they receive the bill at the end of the period! How can a company run a business that way—especially if they need to quote infrastructure costs as part of their bill of materials (BOM) for projects they are providing their customers? They can’t. Not easily.

This problem prompted LucidLink to embark on a quest to eliminate egress from the equation.

The original LucidLink pricing model was “bring your own storage”. This allowed us, as a company, to avoid the problem but placed the burden squarely on our customers’ shoulders. Not only was this not entirely fair, but we realized we were in a much better position to address the issue. 

We looked at scores of providers, from the large hyperscalers, to on-premise solutions, boutique clouds, and decentralized storage services. While we liked many of the alternative pricing models, it was undeniable that the magic of LucidLink works best with the impressive scale the hyperscalers provide. But we found hyperscalers to be the least flexible when putting together special pricing for the new paradigm that LucidLink represented. 

“After all,” we reasoned, “LucidLink keeps the source of truth in the cloud. We are only streaming access to files or portions of files.” The very nature of our service does a lot to mitigate the amount of egress generated, especially when compared to sync methods of other vendors or click and download entire files. We tried to make the point that our customers were not only keeping the data in the cloud but also implementing hybrid workflows because they could easily access the same data from the edge and the cloud services.

Storage providers didn’t buy it. Maybe they grew accustomed to the egress revenues they collect, perhaps it was the inertia problem of a large organization, or even that we remained a small player in cloud file services space back then. Whatever the case, we could not gain a collaborative positioning with them.

We decided that we would adopt an “insurance model” approach. That is, we would leverage the entire pool of data we were putting in the cloud. This allowed us to arbitrage the egress price, bundle it with our service, and sell it at a significant discount to our customer base. While this worked to a degree, the problem was that doing this on our own. We just could not make a big enough dent in the egress price to make a significant difference for our customers. We made it cheaper, but that didn’t change the model.

Enter IBM. Many people are surprised that IBM has such a large cloud business, but the fact is many of the other providers pay them royalties for the object storage patents they hold through their acquisition of Cleversafe. IBM cloud storage is one of the best. More importantly, they were ready to listen to us as we pitched the issue and our idea for solving it. 

We rolled out our first version of the IBM-based LucidLink service over a year ago. The result was egress at a fraction of the cost of other providers. Combined with the efficient way our service accesses files, we saved customers significant money and they loved it. 

But…

We had not yet fulfilled our goal of eliminating egress as a component of the cost model or the fact that it was unpredictable and charged after the fact. 

Until now.

Introducing LucidLink Advanced Filespace for high-performance workloads

A single per terabyte (TB) price that includes egress and is based on IBM Cloud Object Storage on the back-end. It is a no-compromise offering that is easy to plan for, easy to buy and has no hidden costs. 

The stratospheric growth LucidLink has experienced over the past two years is a testament that the technology works well and delivers significant value. And now, with our new egress-included Advanced Filespace offering, we’ve addressed another major challenge our customers face regarding predictable pricing. 

We won’t stop here in our quest to lower costs and make them predictable. Our continued collaboration with key partners on several initiatives will further reduce costs, automate workflows, and eliminate egress. 

What about the tech consultant at the beginning of the story? We hope that he is following our progress and we thank him for laying down the challenge that inspired us to achieve what we have to date. Stay tuned – there is more to come!