An observed increase of uncaught (sometimes high-impact) defects, a request and offer to expand test cases for future releases

Hello,

I really do not want to be the “negative guy” here, so please take the notes below with the understanding that I’m writing this strictly with the intention to spark a discussion and to assist the Bricks’ team (and their users), this is not meant to be taken as venting/finger-pointing at all. I may also be wrong in the sense that I currently believe in seeing many more occurrences of these, it may be more of a coincidence but it just feels like I’m running into more and more defects very recently, and seeing more reports too in the forum for the 1.8.x series (including a high impact seen right away by users in 1.8.2 released today).

I have noticed recently in the past few releases (mostly 1.8.x), that there have been an unusual uptick in what I’d call high-impact bugs/defects, or bugs which should have been an easy catch with automated testing (ideally).

I work for a software development company as my full-time job, so I definitely can appreciate that there will always be bugs / issues with every single release, it’s naive for anyone to think otherwise IMO. However the high-impact (and even the smaller but easy-to-find) bugs should ideally happen with less frequency if there is proper QA / testing in place. This makes me wonder if the test cases currently used need to perhaps be improved in a big way. It’s probably even more important as of recent too as Bricks continues to grow so quickly with a wider range of users and use-cases.

For reference, this is not a complete list at all but is a short-ish list of the defects I feel to be either quite high-impact / or otherwise was just very surprised to see uncaught before release, all filed in the 1.8.0 - 1.8.2 release cycle):

My proposal is one that is probably already well known by the Brick’s team but wanted to at least write it out for everyone’s sake. The proposal: expand the automated test cases, and perhaps some of the manual ones too. Examples:

  • Ensure tests are run with Inline styles set in the Bricks Settings, in addition to External Files of course.
  • Ensure that there is testing to validate the greyed out (inherited) values shown in the builder to ensure they’re showing the expected value.
  • Ensure tests include a comparison of what the builder shows versus what’s seen in the frontend.
  • Ensure that when adding properties to an ID AND class, the changes are apparent in the builder.
  • Ensure that upgrades are performed in a few different types of environments (different hosts, different WordPress and PHP versions, etc).
  • Ensure expected behaviour when duplicating elements with custom CSS
  • Ensure expected behaviour when any items are set in Global Theme Styles, ensure all settings work as expected (especially on top 10 most often used element global settings).

I am happy to help brainstorm further ideas too, or offer any insights I can provide on test setups (although I suspect you’re in the best position already to do that without me, haha). And I realize setting up these test cases are much more difficult than many realize, so I don’t expect this overnight at all, but maybe it can serve as a goal for perhaps test cases that may not have been considered yet.

I really don’t mean to be a pain in anyone’s side, I’m writing this out more to spark a discussion among the Bricks’ team, I don’t even really expect a response to this at all, it’s just something I wanted to share my thoughts on and hopefully the outcome is an improved testing / QA lab setup with expanded test cases to improve the quality of each release going forward.

Thank you for everything you do - it’s nothing short of amazing!! I’m so happy we have a great WordPress theme builder such as Bricks in the market. :smiley: Thank you for your time and your commitment to your users.

15 Likes

Adding this new high-impact bug to the list as well from 1.8.5: Critical Images Error in 1.8.5 (picture tag)

1 Like

Thank you for taking the time to write this up. I don’t have anything to contribute at the moment. If @thomas @timmse @itchycode have eyeballs on this, I think that’s a very good thing.

Thanks again.

1 Like