There comes a time in the life of each software engineer when it’s necessary to do some load/performance testing on the software you have developed, especially when said software is still in development. Usually, you either do it because someone complained that your API is slow, you want to test it under heavy load or check what will be the performance bottleneck as you increase the load. Using AWS autoscaling groups allows you to easily check how well you can scale your application by
throwing more money at it running it on more servers. At some point you’ll probably want to quantify the performance of your app and here are some open source tools available for this.
Apache JMeter is a very powerful and flexible GUI tool for load testing. It not only lets you test your HTTP APIs, but also databases, FTP servers, etc. The GUI allows you to create tests plans and collect data in various different formats. For me, the only missing feature was ability to run it from command line without having to create a test plan beforehand.
ab is a simple HTTP benchmarking tool from Apache. It has really easy-to-use CLI interface and it gives you all the useful stats at the end: mean request time, standard deviation and percentile times. The latter is very useful if your API has SLA regarding response times to certain percentile of your requests. However, ab has one quite important limitation: it only allows you to test one URL at a time, which is not sufficient if you want to simulate real traffic.
siege is very similar in its CLI simplicity to ab, but allows you to specify a file with URLs which you want to hit in order to simulate traffic. This worked really well, but unfortunately the output given by siege doesn’t give you all the necessary stats. The bit which I missed the most was the percentile distribution of request times.
wrk was the tool I settled for eventually. It has a simple CLI interface like ab and siege, but also gives you more control of request generation and final output by exposing a simple API in LUA. By using a short script you can use requests generated beforehand:
You also have access to the data of all the performed requests should you want to calculate the data not provided by wrk directly.
All these tools have their own pros and cons, but eventually wrk was the one which provided me with all I needed in this case. If you have experience with other load testing tools and think they should be on the list, drop us a line.