Runscope API Monitoring    Learn More →

Tutorial: Continuous Integration with Codeship and Runscope API Tests

By Heitor Tashiro Sergent on .

Codeship's logo

A part of an engineer's toolbox is a good automation workflow. One of the tools in that box is a Continuous Integration provider, which can help teams prevent integration problems and improve quality control.

Runscope can be used by itself to monitor and test APIs, but it can also be used as part of your CI workflow. We have a sample Python script in our GitHub that can be used to trigger a set of tests in your Runscope account, and change the build status based on their results.

In this tutorial, we're going to show you how to use the sample script with Codeship. We'll cover how to:

  1. Generate a Runscope API access token
  2. Get your API tests Trigger URL
  3. Set up your environment variables in Codeship
  4. Run the Python script as part of your CI test commands

We have also included a section at the end for Codeship Pro users. :)

Let's get started!

Setting up the Python script

You can find the script sample in our GitHub organization:

The two files we're interested are "requirements.txt" and "". For this tutorial, I'm just going to work with the raw links from our GitHub repository.

If you're integrating this in your project, I highly recommend either forking this to your own repository, or adding these files to a separate folder. That way, you can prevent your build from breaking in case there's an update to the repository.

Getting your Runscope variables

Trigger URL

When running this script from the command line, you can pass one parameter to it, which is a Runscope Trigger URL. If you want to run a single test, you can find its trigger URL under your environment settings:

Runscope test environment settings with the Trigger URL option selected

If you want to run all the tests in a bucket, you can find the trigger URL in your bucket settings:

Runscope bucket settings page, highlighting Trigger URL section

Generating Your Runscope API Key

We need a Runscope personal access token to interact with the Runscope API and retrieve the results from our test run.

To get your personal token, head over to, and create a new application. You can just use dummy URLs for the app and callback URL values (e.g.

Scroll down to the bottom of the page, and copy the personal access token value. We're going to use that in the next step.

A Runscope applications settings page, with the Personal Access Token highlighted

Integrating with Codeship

In your Codeship account, select the project you're working with and click on Settings:

A screenshot of the Codeship dashboard, showing the main page for a project with an arrow pointing to the "Settings" button

On the left-hand side menu, click on "Environment". We only need to add one environment variable here named RUNSCOPE_ACCESS_TOKEN. Paste the value that you copied in our previous step for it, and click on "Save Configuration":

A Codeship environment project settings page, showing how to set the environment variable "RUNSCOPE_ACCESS_TOKEN"

Now, select "Test" on the left-hand side menu.

The Codeship environment already comes with python and pip pre-installed. The first thing we need to do is make sure the necessary packages for the script are installed. Add the following command to your "Setup Commands" window:

pip install -r

Note: Remember to change the requirements URL to your fork or local file.

Next, I'm going to add another command just below it to download our `` file (you can skip this step if you copied the files to your project):

A Codeship test project settings page, showing the "Setup Commands" text box with the bash commands to install the script requirements and download the Python script

Finally, under the Configure Test Pipelines header, in the "Test Commands" window, we can run our script. It takes one parameter, which is the Trigger URL you copied at the beginning of this tutorial. So we can just run the command as:


Note: Make sure to change the URL after `` to your tests Trigger URL.

A Codeship test project settings page, showing the "Configure Test Pipelines" text box with the bash command to run the Python script

Checking Your Build Runs

In your next build runs, you should be able to see a new step running the Python script, and hopefully returning a green checkmark ✅:

A successful Codeship build run, with a line highlighted for the command that runs the Runscope Python script

Integrating with Codeship Pro

Codeship Pro is a more advanced offering from Codeship. It offers you more control and customization over your CI process, and includes native Docker support.

If you're using the Pro plan and want to integrate your workflow with Runscope API tests, we created a sample project that you can find here:

It's based on Codeship's tutorial project, and it includes instructions on everything you need to set it up. When you run it, it will spin up a separate container to run your Runscope API tests, in parallel to the sample Ruby application.

Continuous Integration Complete

With Runscope integrated into your CI process, we hope that you have even more confidence in your builds and that your APIs will be 200 OK.

We used Codeship in this tutorial, but these instructions should also apply to other CI providers like CircleCI or Jenkins (we have a Jenkins plugin). If you need any help with those, please reach out to our awesome support team.

Are you integrating Runscope in your build process? If so, we'd love to hear how you're doing it, just reach out to us at or on Twitter @Runscope.

Related Resources

Categories: code samples, tutorial, testing, integrations

Everything is going to be 200 OK®