A Lesson from the Pacific Stock Exchange
They asked if I could come in and spend a day looking at their systems. As a software architect, this was my kind of project. So I said yes. Of course, yes.
It was the Pacific Stock Exchange in 2002 and they were working on an electronic trading exchange to do digital transactions.
I walked into the conference room and spent the next three hours listening to different teams present their portions of this complex architecture and system. Some teams were staff that worked there. Other teams were third-party vendors.
And this was, if I remember correctly, the second or third team that had been implementing this new system.
The approach for the system was based on tiny messages sent around in a distributed environment. It's been known as message-oriented middleware and at that point in time, it was all the rage.
The loosely-coupled architecture meant lots of teams could create their own modules and just drop them into the flow. But that same approach was causing them performance problems.
After all the presentations I asked a single question – one I remember to this day. “How many messages are created across the entire transaction?”
Because this was an options exchange, the idea was that something (an issue, at a future time, for a future price – Microsoft May 50) would go up for sale. The trading platform would then determine who would get the right to make purchases.
Instead of the outpouring of people screaming (like you imagine on a trading floor), the options would be available to 1-3 traders, who would decide how much of the lot they would want.
So I asked, from the time Microsoft's options went up for sale, until the traders finalized their purchases, how many total messages were fired off.
No one knew.
This is the nature of enterprise software development projects built by several teams of people, each responsible for parts of a system that must come together for an end user.
If I had one takeaway from that day it was really this simple: when multiple teams are all involved building a single solution, plan for frustration, disappointment, an inability to do impact analysis, pointed fingers and more.
Plan accordingly.
Why didn't the White House use WordPress?
A lot of people have been passing around an article today from Politico. While I like the idea of using WordPress everywhere we can, I find that the article is misleading.
In the article, there's a comparison of the federal system that has had serious trouble (healthcare.gov) and the state systems that have performed much better – which use WordPress.
What the article doesn't mention is how vastly different the state exchanges are from the federal one. For example, the California Exchange (where I live) is a seriously simple site. I didn't check if it was done in WordPress, but none of my friends would have trouble building that site in a weekend.
The California site basically asks a set of questions (which could easily be done using Gravity Forms). Based on the answers, a conditional response appears with a button to take you to a new page to see plans.
These plans are static entries of data – easily displayed using custom post types. No problem. Nothing interactive there.
If you're interested in getting more information, you're taken to another page, with a form (even easier, a contact form that Gravity Forms would eat for lunch) where your contact info is collected and someone will follow up.
Yes, my take is that if the content were ready and prepped, we could knock that out in a couple of days. And we'd have to hire a designer to make it look pretty. But we'd be golden.
But the federal exchange isn't a magazine or lead gen site. It's a serious piece of enterprise software.
Now, you know I love WordPress. But I also build enterprise software for a living. In fact, I've bid on projects with friends from CGI Federal (some smart folks).
So when people are comparing a brochureware site to enterprise software, I have some thoughts and words. You know what I mean?
How enterprise software is(n't) built
When I talk with WordPress companies that do development, one of the questions I ask, to understand the kind of complexity they face, is the size of their projects.
I don't mean how much money they make. I don't care about that. I mean how many different people are working concurrently on the project. Enterprise software isn't normally built by 2 or 3 people.
One of the pieces of software my team has built is a rule engine that determines the calculation of rent for a family that wants to move into a unit that was revitalized using tax credits, but is also subsidized because of their income. That rule engine has had 5 people work on it. With rules changing yearly based on legal code (500 page books that get published and that we have to read), this isn't a simple rule engine.
And the crazy part is that the rule engine is that it's only one part of an overall system – where another couple of teams are working on the rest. And we all work in the same company, with access to the same code repositories and communication tools.
We've been working on another enterprise api for mobile inspections where we're working with another company. It's been months and months of work. Three companies building one solution.
Enterprise software is complicated not just because of rules and rule engines. Not just because there are several people stepping on each other's toes (and code). Not only because communication across companies is difficult. It's also challenging because it's often custom.
The site in question isn't just a website. It's a rule engine. Several, actually. It requires integration into several different systems. Tons of handshaking. And most of it hadn't been done before.
Requirements are discovered and suggested.
Over time, things appear and decisions are made, and at the size and scope of these projects, you don't get the chance to stop, roll everything back and start over. This isn't a small and focused plugin.
In the case of healthcare.gov, multiple vendors were working on multiple parts of the project. Whether the platform they used was WordPress or not wouldn't change the complexity of multiple teams working together. Whether the source code was licensed under open source or not wouldn't change scope creep.
One difference between the federal exchange and the California one is that the federal exchange added a “create your account” as the first step. My understanding is that the requirement to put that first in the user flow (and process) wasn't in place at the beginning of the project. We all have experienced scope creep. Right?
I know how it feels to get a last-minute feature request – one that's not really a request at all. When your client is paying for custom development, suggestions aren't really suggestions.
More importantly, that first account creation (which slowed a lot of people down) module was written by someone other than the folks who wrote what now is step 2. So there's not only last-minute extra work, but flows and handshaking changes that all have to be taken into account.
The VP at CGI Federal wrote that their modules had passed several tests before going into production. I get that. But I also know that a “mine is working” doesn't cut it when what you really need is “ours are working together”.
And I sit back and wonder, much like I did when I sat in that conference room at the Pacific Stock Exchange. Did anyone evaluate how this would work once all the parts came together?
WordPress is great for many things
I don't have a problem with people saying they should have looked at open source software or at WordPress for the government exchange. Everyone has their opinion.
The issue, after all, isn't the software. Serious engineers can make any software do just about anything (even if the core software is almost unrecognizable later).
The issue is that there is no silver bullet when it comes to building enterprise software – especially across vendors. And when we, in our community, praise others who are championing our platform (and talking about developing enterprise software like we're developing a plugin), we do ourselves a disservice.
I'm excited for where WordPress is going and I hope that we'll work through some of the dynamics that make scalability a challenge for certain kinds of applications built on WordPress (logged in users needing partial cache in membership sites comes to mind).
Until then though, you won't hear me suggesting that healthcare.gov should have used WordPress.