The Goal
There are some great posts out there that compare WordPress frameworks. My recent favorites have been by Clifford Paulick. His part A and part B are the hallmark of robust comparisons. In a conversation with him a few months ago, I wondered aloud about a comparison of their performance. I’ve seen WordPress Frameworks compared by tons of factors including features and price like Clifford did, but I’ve never seen them looked at from a performance perspective.
And you can see why. You’d first have to buy them all, which most folks wouldn’t do. Then you have to design a single layout that all could do – which means you’d have to know how they all work. And then you’d run your performance tests.
But if you did such a thing, you could let people know that – all things being equal – which theme frameworks performed better than others. And that might, just maybe, be interesting to someone who was looking to purchase just one and get to know it well.
The Competition
Before I get into methodology, let me give you some context on which theme frameworks were compared.
- Numer of theme frameworks: 7
- All personally purchased – no freebies to influence my testing.
- One xml export from ChrisLema.com – 157 posts, 583 comments, 33 pages, 6 categories.
- One standard blog layout with a custom menu, and three elements in the sidebar (About me, Twitter feed, Tag list)
- 10 posts on home page
- Genesis
- WooThemes (Canvas)
- Headway
- Catalyst (with Dynamik as the child theme)
- Pagelines
- Ultimatum
- iThemes (Builder)
Comparison Methodology
What I did was jump over to my playground (wpadvisor.com) where I do all my testing and I created seven sites on that multisite site. So each site was stand alone. And in each site, I imported the same file. I even imported images so they’d be local. After the import, I deleted the initial post and page, and configured the menu to show the core set of categories. From there I configured the sidebar to have the widgets that I mentioned above. Each theme had different requirements to get all that configured, but none of them took longer then 20-30 minutes to get everything done.
Then I hit each site multiple times. Because I wanted to make sure none of these sites would do their first load during the test.
At this point I should mention that no caching plugins were installed because I host my playground on WP Engine where things move fast without them. I should also note that the page speeds here may not match your own experiences if you’re hosting your site on crap solutions by companies rhyming with MoBaddy.
At that point, I took the home page urls and feed them to the Tools page at Pingdom.
I captured the data and pulled it into spreadsheets for comparison. So here’s the info you’ve been waiting for.
Comparing Request Counts
| Framework | # of Requests |
|---|---|
| Headway | 12 |
| iThemes Builder | 15 |
| Genesis | 19 |
| Pagelines | 21 |
| Catalyst / Dynamik | 22 |
| WooThemes (Canvas) | 25 |
| Ultimatum | 28 |
As you can see, the spread was wide. I would have guessed that it was much tighter together, but that wasn’t the case. That said, it only matters in my mind if there’s a relationship between the count and the load time. Which is why you’ll see how some of the larger request counts still did well in terms of speed to load.
Comparing Load Times
| Framework | Load Time |
|---|---|
| Headway | 226 ms |
| Genesis | 964 ms |
| WooThemes (Canvas) | 992 ms |
| Catalyst (w/Dynamik) | 1.2 s |
| Ultimatum | 1.22 s |
| iThemes (Builder) | 1.26 s |
| Pagelines | 5.18 s |
Headway blew the others away – which may have been helped by how few requests it made. But WooThemes loaded up quickly despite it’s request count. Falling to the bottom was Pagelines, but that wasn’t surprising when you see the next table (looking at page sizes).
Comparing Page Sizes
| Framework | Page Size |
|---|---|
| iThemes (Builder) | 299 kb |
| Headway | 1.1 mb |
| Genesis | 1.1 mb |
| Catalyst (w/Dynamik) | 1.6 mb |
| WooThemes (Canvas) | 1.7 mb |
| Ultimatum | 1.8 mb |
| Pagelines | 5.2 mb |
iThemes and Headway jumped in front, but many of the others were right behind them. Except Pagelines, which was a bummer.
Looking at Pingdom’s Performance Grade
| Framework | Pingdom Perf Grade |
|---|---|
| iThemes (Builder) | 100 |
| Headway | 98 |
| Pagelines | 96 |
| WooThemes (Canvas) | 94 |
| Ultimatum | 91 |
| Catalyst (w/Dynamik) | 90 |
| Genesis | 84 |
Any time you get a combined score from anyone, like Pingdom, it’s worth digging into how they’re creating their algorithm. I confess, I didn’t do the digging. I may later. But that’s mostly because all of the themes had a nice high score. I’ve seen sites that have far lower than 84 in the total Performance grade.
One other thing to note is that many of the factors that affected the performance grade could have been solved by minifying static files, combining them, and using a CDN. But I wanted a straight comparison, so I left things native.
Conclusion
As always, your selection of a “winner” is based on what you value most. Not every theme here was a drag and drop framework – and maybe you care about that more than anything. Some are easier to learn than others.
In terms of pure performance, Headway seemed to run away with things with iThemes in second place. Genesis & WooThemes were right behind, when I created an amalgamated score.
What does that tell us? Well one interpretation would go like this:
The players that have been in the game longest have had the time to really optimize their frameworks for performance.
You might have a different take. That’s on you.
Fantastic article. 5.2mb’s for Paglines – wow, what a whopper. Always a danger that sites become bloated, so good to start small. Great to see the speed differences too – thanks for doing the leg work for us. Going to stick with Canvas for the foreseeable future.
I’m an Builder gal myself, started using it a year ago with very little experience with WordPress and php and it has really helped cut down the time spent learning how to design WP sites. I just found your blog through twitter and I really love the clean simplicity of it, I’m curious, what do you use for your site?
I switch between multiple frameworks but with roughly the same design. Right now I’m using Catalyst.
Great testing methods Chris. I’ve got a couple of WP installs. If they decide to go deeper, I’ll let you know.
I find it hard to believe PageLines served up 5+ Mb of data when the entire framework package is less than 1.5M, where is the test site so we can see it?
I think the test is bs. PageLines Framework is only around 1.2mb so how can you ever get 5.2mb without uploading extremely large graphics?
Give us the Pingdom Tools links so that we can verify the results you got.
Also, why would you want to setup multisite where you have all the frameworks running instead of replacing a framework on a singlesite?
Not quite sure how you’re getting 5s wait and 5mb pageload for PageLines? Where is the consistency in your evaluation? Where are the results? Where are the benchmarks?
I just loaded up my personal site, and am getting a 2s load with a weight of 564 kb on PageLines.
http://tools.pingdom.com/fpt/#!/pnbEAZihN/http://nickhaskins.com
I’m excited to see that my favorite theme has pulled ahead! I love working with Headway, and ever since the 3.0.5 update it’s become blazing fast and hyper optimized. Awesome comparison
Really interesting. Props to you to take the time (and budget) to do it.
I’m wondering how much real-world results may vary. For instance, most of this frameworks come with a lot of custom widgets. I’ve experienced myself how some of those widgets are total performance killers, but I guess people use most of the functionalities the frameworks come with.
So, when you deploy all the particular features for each framework (widget, filters, settings, etc), instead of using a simple template like yours, how do they compare?
To the folks who can’t believe my results, who want more proof, who disagree with my approach, or who missed the fact that on chrislema.com I use a 600×400 image for every post, I don’t mind approving your respectful comments and disagreements.
The 5 images add up to 916kb, so where the other 4.5Mb coming from? Are all the themes/frameworks set to show the image/full post on the blog page?
It was 10 posts. Not 5. Only titles, excerpts, and images were on the page. I’m positive that some images were larger than others, but all were resized to the 600 width I specified.
Odd, your main blog only shows 5. So did you set all the other themes to show the images too?
Yes. All themes were set up identically – with 10 posts. See the beginning of the article.
So all the themes have the same images ya? With that being said, they should all weigh the same (the images). Obviously the themes in your test aren’t setup the same way, otherwise, all frameworks would be consistent in the weight of the images.
Your ranges are completely out of the realm of possible. 5+mb for PageLines, compared to all the other themes that only show 1mb. It’s obvious that those other themes didn’t load the images, or we’d see the weight of those images in your results.
They were all set up the same guys. Try your own comparisons and write your own review and I’ll link it from here if you like. I get that you’re incredulous, but I didn’t have it in for Pagelines. I like the theme and was surprised. I’d guess it was related to image processing but it’s just a guess.
Well as there are no links to the pingdom pages I guess we are all just left guessing.
This is great information. Thank you Chris! We are currently using Genesis and have been sitting on a developer license for Headway. Now we are going to try it out.
I am a Hybrid Core guy myself.
Awesome post Chris. I’m not surprised to see Builder and Canvas at the top but happy to see them there as they are my go to themes.
Btw, how would you change your scoring of the different frameworks in your other comparison post now that Builder 4.0.x is out with responsive support? Personally, I love how flexible it is and how you can set your own cutoff sizes in a simple function in functions.php. I love Chris Jean’s vision of evolving the framework…
Thanks again for your very informative posts!
Hey Chris. What a great comparison review. Thanks for including Headway in your test subjects too.
The true winners in any review is the WordPress community. Isn’t it great the community has so many choices of themes and frameworks to pick from. No one theme or framework is the answer for everyone. And having all of these choices for the community is the win-win here.
Thanks again Chris for putting in the time and effort to do this review.
Thanks so much for doing this..It was one thing that was missing in the Paulick reviews…:)
Awesome breakdown man. Bookmarking & saving for reference.
I’m confused; you say in your conclusion that Headway comes out on top, but numerically Builder is actually on top both times… what’s the rationale for that? Thanks for this great review,
A weighted ranking based on position put it in second. But it’s still great.
Could you look at http://swiftthemes.com/ and test that one as well? They claim to be the fastest. Even faster than Headway! Is this true?
I may check them out. I’ll let you know.
When I first checkout Headway 3 I was hesitant, as I feared the amazing flexibility and drag & drop editor would hamper the performance.
But to my suprise, Headway is super optimized, fast and lean. Since then, I’ve been a huge Headway fanboy =)
Good to see this test Chris, as it will prove to many potential Headway users that Headway indeed is among the top frameworks in the WordPress community.
@chris I run PageLines on dozens of sites and my customers who upload huge graphics dont see load times like that on WP Engine.
I’d like to run the same test on a single instance and not multisite on WP Engine. Would you mind sending me your xml file?
I think we’re finding the potential issue. I think it’s related to how Pagelines deals with images. I not only have large images I use. Sometimes I upload a really large image but use one of the pre-generated sizes. I’m checking to see if one of those images was loaded by Pagelines and sized on the page rather than before loading. That would explain it.
Sounds feasible. Still, that would be a very ineffective use of images. Potentially expensive, hosting-wise as well.
No doubt. That said, my hosting provider is awesome, sometimes I’m careless on the run, and the other frameworks seemed to handle it, suggesting at least one place to improve or one warning to end users.
Chris, PageLines by default shows the featured image thumbnail, like any other theme. Also only shows excerpts on the main blog page by default.
Unless you changed the settings to show the full image..?
Hey Chrislema sorry for the previous message. I just found your no hold barred review on Thesis 2, and you’ve made your point pretty clear! Loved the comments as well! I personally don’t mind Thesis 2 as I can code css to make further customizations. But will be giving Ultimatum a try as well! As far as calling Thesis 2 a drag and drop framework, I don’t think Thesis 2 qualifies for that!
Thanks for your work man!
I’ve been using Builder since it came out, and love it. In fact, I now use it exclusively for new sites, but a few of my sites are very slow and I was afraid it might have to do with the theme. Thanks for doing this study – now I’m not worried about having to learn a new framework!