PHP Office Hours with Chris Hartjes: How to Get Started with Automated Testing, PHPUnit, and BDD
Codementor PHP expert Chris Hartjes is one of the best-known automated testing advocates in the PHP community.
A huge consumer of open-source software, Chris tries to give back to the community via his blog @TheKeyboard, by speaking at conferences, and by co-organizing the GTA PHP User Group. He is also a big believer in the power of testing and automation as secret weapons for organizations to deliver high quality applications quickly.
Chris recently sat down with us during Codementor’s Office Hours as an open Q&A session for viewers to ask him about testing and PHP.
The text below is a summary done by the Codementor team and may vary from the original video and if you see any issues, please let us know!
So, here’s a little about me: I’ve been doing programming on the web for 16 years. I got into testing because I had the experience of working on a horrible code for an adult dating website, where I had to work overtime for an extended period, and I never wanted to do that again. I discovered extreme programming, and through that, unit testing. The landscape for testing was much different back then, and PHP unit wasn’t even available in a public release. There were very few open source web app frameworks and testing tools, so I taught myself a lot of things.
The current state of testing
We’re at a point where we have too many choices for tools. Sometimes too many choices is bad, but if we look at the type of testing to do, we do have a couple of options.
If we want to do unit testing, PHPUnit is still the tool I’d recommend to everyone who wants to get into testing. Unit testing actually gives you the biggest return for the amount of time you’re going to spend on it. It allows you to focus on your code and the module level. An analogy I’d use is lego blocks. You snap lego blocks together and create awesome applications.
If you’re interested in behavior-driven-development, Behat is good. I use it to do automated front-end testing of applications, because you can hook it up to a browsers (as a lot of things such as Phantom.js depend on browsers), and write up a test to simulate entering values into a form, logging people in, and other things a user would normally do. Once we get it past unit-testing, and using a functional-type testing with Behat, a whole bunch of tools out there are hybrid tools and not really fully behavioral style for the unit (e.g. Codeception, PHPspec, Atoum). So there are a couple of options if you don’t like the unit testing style.
I think this preference may have to do with the generation of a programmer, what with all the college/university cycles. We’re now at the third generation of programmers where they have testing as an option shoved into their faces because the tools are all available. So I’ve seen some changes, like the adoption, but there is still a lot of work to do, still lots of advocacy which I spend a lot of time doing. There are still a lot of tools and skills developers are not being taught to make their life a lot easier as they try to figure out how to do testing.
I have never written a unit test and end up pushing hotfixes all the time, but we run a really big project right now and I keep thinking we need to change toward a new setup. I don’t know where to start. What kind of advice would you give to someone to get into BDD or even just PHPUnit?
There are two interesting things you’ve brought up, where you’re stuck in the cycle with users testing your stuff for you, and you creating hotfixes in response to the users. You’re not alone, since there are many applications that use users as free QA people. It’s okay if the app is new, but the longer since you’ve created it and the more people who use it, it will become more critical to their success, and eventually users are going to rebel and realize they’re being used as testers, and you may see a significant amount of people who will not want to use your stuff anymore, or worse talk shit about your product. It’s a fine line to walk, so I understand the constant pressure.
These days I spend a lot of time looking at processes and getting people to change what they’re doing, so I know it’s a very difficult cycle to get out of when you are pressed for time with a lot of features you need to get done. What I’d suggest is do these things in parallel—you can keep doing hotfixes while taking some time to write the tests.
Developers aren’t necessarily taught how to write tests, and in my experience it is extremely difficult to learn by yourself, so you need to find people who already know how to do testing stuff. And that can be difficult because generally speaking, all those people are like me. They have full time jobs and commitments and their ability to consult or mentor you during regular business hours can be limited, so I definitely feel your pain where you don’t know where to start.
If you already have the staging and development servers, you’re ahead of 90% of the shops out there, but the thing you’d need to do is get these things automated. What you don’t want to do is manual testing—you’d want a computer program to do your testing. If you have an app that’s in a state where you don’t have time to go back and write a unit test for, there’s definitely things you can do to write automated tests to test the flow of the application and test the most critical things. What you’d end up with is a test that reflects how people are actually using your application, and you’ll know instantly when something is busted out of your regular user flow. The typical flow is to make sure users are always able to log in, sign up, use the functionality. This is especially critical if you’re doing some e-commerce thing, and there’s a reason Amazon sat down and figured out how much money they lose every minute they’re down. The ability of people to not use your application is going to cost them money for sure, and it might cost you money.
In conclusion, I think it’s worth investing some time looking at Behat + mink, especially if you have an application that’s already up and running, providing value, and people are already using it so you can’t afford to have a catastrophic fail. If you combine the two in your use, you can write all these kind of tests to simulate a user logging in, signing up, and so on. Of course I can be biased and tell you to read my books, as it will help you create the foundation for unit testing, but I know there are still some work to do to show people who’ve never done it before the best way to do testing.
If you’re starting from scratch, how do you take a look at your current setup and find out what are the most important things to do?
I tell people once you get whatever testing framework in place, it means they’ve made the commitment to test. The first test to do is to make a test and prove a bug someone else has reported to actually exist. You can say from this point onward, every single bug that comes in, you’re going to write a test to verify that the bug actually exists, and we’re going to use that test to refactor our code until the test passes with the correct behavior. You have to figure out how it’s supposed to behave, write a test that expects the proper behavior, and keep working on your app until the test passes. To me that is probably the method you’ll find to have the most impact. Sure, you have the pain upfront to learn how to write the test, but instead you’d end up with a test suit that actually reflects the problems that the app actually has. You can use code coverage tools and have a goal of a 100% code coverage, but with legacy app that’s pretty much impossible and a really big waste of your time. So pick a bug people have reported and write tests until they pass is probably the most impactful thing to do when you’re just starting it.
Once you’ve tested your bugs, is there a best approach or a best path to follow to start applying and writing tests for other components of the application? (E.g. Look for things like: What if this fails causes the most damage to/be the most irritating for a customer, or cause the most headaches for the admins later.)
I tend to recommend to people to work on the bugs. The next step will be for everything new that you do. We sit down before we assign the work to someone—nothing gets worked on without a ticket of what needs to be done, and the person who wants work done needs to say what needs to happen for them to accept the thing. What you’re trying to do is get people to write testing scenarios for you, and you can turn those things into tests for whatever framework or tools you happen to be using, and then you write those tests first and keep writing code until those tests pass. Get all the bugs taken care of first, then writing test for anything new, and then you’d have to be really ruthless with your code reviews when pushing code to staging and staging to production.
I tend to be the sole developer of the project or only involved with one or two other people, so you don’t have all the steps you might have in a larger company for working with a large team. How do I go about testing?
Time pressures are time pressures, and deadlines don’t change. A lot of it comes down to how much ability you have to push back and say that you can’t get things done because you’re not 100% the fixes are going to work. I typically don’t work for large companies, so I am familiar with the whole process of pushing back and saying the time pressure is causing you to ship things that aren’t perfect. So you can present them with the give-off triangle with the fast, good, or cheap where one thing has to be sacrificed. The people who are above you are the ones who have to be OK and those are the ones you always have to push back on. In my experience is if you’re reasonable with them and show what you’re trying to do while highlighting all the time you take and things that broke because you didn’t have time to check it properly. So there are ways to make the case to say things are getting busted because the deadlines are pushed too hard and you’re juggling too many things, and everything is only getting out at 80% done instead of a smaller subset of things 100% done. Only incompetent managers would say that slowing down to get things right is the wrong thing to do. I mean I get it, but I always try to have a say in the scope of work and the deadlines as well.
I’m starting work on a project where my role will be to write automated tests. It’s new for the company. They want to use Selenium, but I thought PHPUnit was the de facto way to do automated tests. Where should I stand?
For those who don’t know, Selenium is an add-on for web browsers that allow you to record your interactions, or sessions of what you’re doing. You can then play them back and watch what you’re doing and you can see if errors happen, since the browser basically repeats what you did…at the same pace you did it. It’s an interesting tool to look at the UI of your application, but the problem with using Selenium is that it’s not sharable.
There is no easy way to move those Selenium test you’ve written from person to person. I’ve seen companies twist themselves into knots with remote desktops and shared folders online and try too share between the testing team the scripts Selenium saves. I find companies spend more time trying to keep that stuff in sync than go about the process of writing tests. So, Selenium covers one type of thing. However, just because the application itself is working correctly, it doesn’t handle integration tests, unit tests, and it’s not sharable. Tools that use Selenium as an interpreter still can’t be taken and imported into a Selenium script. So if you want to use Selenium, I’d suggest making it a teeny-tiny part of your testing strategy. You’re far better off investing time at the code level and making sure your components and application work correctly.
What’s best to test and not to test? And do you have any advice for when the person writing the tests is different from the person writing the code.
If people really want to do the UI testing, I highly recommend you write tests for the functionality you find to be the most critical. Logins, user signup, and if you do e-commerce things, always make sure users can buy things, and if you have a dashboard where users log into, those things have to work. Basically, anything that would have catastrophic effects to your business if it’s not working—that’s what I would tack off first. Then it would be a confidence thing, where you no longer have to worry if a push is going to break the front page of the website because you’ve ran all the automated tests so you’d know instantly if you’ve broken something. This is all about moving the cost of the bugs to the cheapest part of the development cycle.
I always think it’s a multiplier thing in terms of the cost of the bug:
- Bugs found by developer when writing the code = 1x
- Bugs found by somebody else = 2x
- Bugs found in staging = 5x
- Bugs found in production = 100~10,000x
You could have a bug that destroys your company, so definitely try to get these automated tests down before production.
No, you can’t, because you have the overhead in your test when you’re spin up your browser. Even if you’re using a headless one, automated UI testing is expensive in time. You can have a test suit with thousands and thousands of unit tests that can run a lot quicker than one single test that is using a browser
Is there a way smoothly integrate testing into a pre-existing project? And generally speaking, how do you get in the mindset to write code that is compatible with automated testing?
There is no way to smoothly integrate. Generally speaking, the architecture of an application that was not written with testing in mind will push back and make things harder to implement. Any architectural decisions you made that makes it difficult to inject dependencies in would make it hard to test (e.g. using a lot of registered globals). My approach is to get my testing tool of choice set up, then I use Xdebug for code coverage, and then I’d ask my clients what the most critical things are for the application and then get started on writing tests for those. If we do have existing tests, then I’d have extremely low-level code coverage and I’d start trying to fix those.
Can you talk about Travis.ci, and do you use it?
Travis.ci is an integration server which works really well with Github. Anytime you push your code up to github, you set hooks up so that a notification is sent to Travis.ci systems. They basically spin up littler virtual machines and they will check out your code. I have an open source project called OpenCFP (which calls for papers app for conferences), and I use it to run on my tests. Again, it’s automating something for you so you don’t have to worry about it any time a code gets pushed. If something gets broken, I find a email that says something is broken.
What’s the differences between PHPUnit, Behat, and PHPSpec?
PHPUnit is for Unit testing, or a test for a specific unit of code or object. You want to make sure you can set dependencies for that object, and you write tests to verify that code behaves the way you want. You can use it for integration tests, but those capabilities were built in afterwards, and PHPUnit is best for unit testing.
Behat is more behavior driven, where you’re writing the tests in a language that’s closer to spoken English, and the framework translates on your English-like tests into commands Behat runs. Behat has a lot of built in ones for very common testing things, but Behat will also prompt you to write things that don’t exist. It’s for behavior, so you’ll be trying to verify the behavior of your application. Think of it more as integration tests as opposed to trying to focus on an individual unit of code working.
PHPSpec is similar to Behat where it uses specifications to do the same kind of thing Behat does. It also turns pseudo-English such as “I’m expecting the code to behave in this sort of way” into a command the framework should run.
In all three tools, you’d write the test cases, the methods, and the code that should support you while you tests.
Is there an easy way to run the PHPUnit test within the browser?
Nope. Command line is the way to go if you want to do really effective testing. Simple Test was the first testing tool I’ve worked with, and it only worked in the browser, and I think the problem with it is it became really difficult to run in an integration environment. This is why PHPUnit is so good, because it’s built for the command line, so there are no hoops you’d have to go through to get it to work.
When working with multiple API’s do you skip certain tests, or do you need to go get mock responses for every possible API call / scenario and write tests for both success and failures?
Good question. A lot of it depends on what your goal for your test is. If your test is for critical stuff and you want to use your API. If we start at the high level, you’d want to test every response and test as many combinations that are critical (e.g. one of your API calls.)
There are no black-and-white rules for this. Basically, you’ll be looking for enough tests that will give you confidence that the thing isn’t broken. It will also give you confidence that if you change something else, the test will tell you that you’ve accidentally broken something somewhere else. Also keep in mind we’re trying to move the cost of bug-fixing to a cheaper part of the development cycle.
I have learned to write the test scenario first, then write the functions. Thoughts?
That’s test-driven development, which is my preferred method of doing things. If the testing framework is set up already when I write code, I definitely write the tests first and then write code until the tests pass. I think that’s the best way to do it when you already know what you want to do. Often I find that I’d first write prototypes for some things and none-final versions of them. Once its proven I can implement the idea I was working on, I go back and write all the tests for all the things I want to do with that particular bit of code, and I’ll keep writing tests until everything passes.
A little self-promotion here, but if you’re interested in learning the foundations of PHP unit testing, you can read my book, The Grumpy Programmer’s Guide To Building Testable PHP Applications, which covers how to refactor your applications to make them more testable. I think beginners will find this helpful in learning how to get into testing, since it takes some actual bad PHP code and fix it up. I have been contemplating about creating a second edition of the book, since I realized there’s so much stuff people don’t know or are having a hard time understanding.
If you want to learn more about how to use PHP unit, I have the Grumpy Programmer’s PHP Cookbook, which mostly covers the basic functionalities of the PHP unit such as how to install, configure it, and set it all up. If you go to http://grumpy-learning.com/ you can see some of my screencasts, which are for sale, which consist of real examples of how people do things. I’d highly recommend the PHP testing bootcamp, even though it’s $100, if you’re a freelancer and you learn these things, you’d make back the money you spent in about two hours, so it’s not a lot if you’re looking in the grand scheme of things.
Seems like grumpy programmer is not that grumpy at all, no yelling and no bad words!
A long time ago I made a decision to create a brand to put in all my testing efforts, and the brand was the grumpy programmer. For those who are interested in their own content, marketing and consistency matters. I have what I consider a fairly entertaining twitter account, but in real life I’m nothing like the person on twitter. If I was, I would be unemployable, so certainly in my day job I don’t do a lot of antics I talk about on twitter.
Also, thanks to everyone who set aside the time to listen to me talk about testing and to ask me questions. You can hit me up on twitter @grmpyprogrammer or e-mail me anytime on anything. I read every email that gets through my spam filter, and although I can’t guarantee I’ll answer every email, I’m a huge believer in mentoring and passing on the knowledge I’ve learned so they don’t have to go through what I’ve experienced. If you really want a grumpy rant, here’s one: If you’re working in a company that doesn’t believe in giving you time to do testing, remember that programmers are in high demand and you don’t have to put up with that sort of shitty company.
Other posts in this series with Chris Hartjes:
- PHP Tutorial: How to Test Message Queues and Event-Driven Systems
- PHP Testing Tutorial: Mocks, Doubles, Dummies and Stubs