Runscope API Testing and Monitoring    Learn More →

Integrate Runscope API Tests into TeamCity Builds with New Plugin From MANGOPAY

By John Sheehan on .

Hugo Bailey recently reached out to us to let us know about a new project his team created to include Runscope API tests in their TeamCity build pipeline. The plugin was developed by Bertrand Lemasle and is available on GitHub. If you use Runscope and TeamCity, be sure to check it out!

We were eager to learn more about how MANGOPAY uses Runscope to keep their API healthy, so Hugo and Bertrand agreed to an interview we'd like to share with you.


MANGOPAY is a payments API. Unlike traditional payment solutions, we provide our customers with a fully customizable technology.  The API enables websites such as marketplaces or crowdfunding platforms, to accept payments, hold the funds on segregated accounts, and manage the pay-outs. Thus, it’s a truly end-to-end payment solution. The API also has various usual features to handle KYC, chargebacks (and disputing them) and detailed reporting.

Our solution is ideal for marketplaces, crowdfunding platforms and the sharing economy. Our clients, either in-house developers or external agencies, can integrate our white-label API directly into their solution. We offer a free sandbox environment where developers can fully test the API. We also provide SDKs and kits in a variety of languages to facilitate and speed up the development process.

What led you to build the TeamCity plugin?

We aim to have one major and one minor release per month. The process of shipping these releases is fairly standard - in each case, it’s developed in our integration environment where tests are continually run to ensure everything is going smoothly. Then, once everything is completed, we’ll publish the release to our internal testing environment whereby the product managers can run further non-regression tests, and of course specific tests for each development that was done.

We rely a lot on our API tests. In addition to unit tests, we use API tests as non-regression and functional tests to prevent any unintended outcome during our development process. In the past, we used a homemade tool in conjunction with a unit test framework that allowed us to write, test and chain multiple API calls. While functional, this setup was not ideal as it required a lot of time to setup a complete test correctly. Moreover, this tool was tied up with our API and we spend hours just to keep the tool up to date.

Runscope seemed the right fit to replace this. But to be usable in our continuous integration, we needed Runscope tests to be understood as actual unit tests. And it was this that led us to create a TeamCity plugin.

As TeamCity now understands every single step of each Runscope test as a unit test, we simply replaced our unit tests with Runscope tests. Runscope is now fully integrated into our CI, and from a developer point of view, almost nothing has changed. We even have a more detailed output now than before thanks to the Runscope assertion details.

What are your plans for the plugin going forward?

We would really like to improve the plugin and offer more features. Today the plugin is far more powerful than we first imagined but that has made the configuration harder. For example, we would really like to make the configuration easier by making it possible to choose buckets and tests from a list rather than supplying raw IDs.

Improving plugin stability seems a priority too, given the troubles an unstable CI process can bring along.

We’re not using Java in our day to day programming languages. That means that our plugin could be somewhat clunky and there is nothing we’d love more than to see people get in touch with us via our the Github repo and start to contribute and improve the plugin within the weeks or months to come.

Our thanks to Hugo and Bertrand for taking the time to answer our questions. If you're interested in helping contribute to the plugin, be sure to head over to the GitHub project and participate. If you're new to API testing, learn more about Runscope's cloud-based API testing solution or sign up for your free trial account.


Monitor Your Webhook Workflows with New Incoming Requests for API Tests

By John Sheehan on .

API testing and monitoring has traditionally focused solely on outbound requests that simulate what a client of a particular API or endpoint would experience. But APIs are increasingly becoming a two-way conversation, with data flowing into apps asynchronously based on events happening (e.g. GitHub commits, Twilio text messages received, bots receiving requests, etc.). The most popular way this occurs is via a pattern known as webhooks.

Testing a webhooks implementation can be difficult. While you can exercise specific endpoints that consume an event payload (Runscope is great at this), generating test events that mimic real-world conditions is a manual process. What we need is an API testing tool that understands data going out via API calls, but also data coming in via asynchronous callbacks, and the relationship between the two.

Introducing Monitoring and Testing for Webhooks

Today we've added a new feature to our API testing and monitoring lineup that allows you to combine outgoing API calls with incoming callback requests. To understand how this works, let's take a look at a simple example.

Testing Twilio Status Callback URLs

Twilio is an API that allows developers to add communications (voice, text, video) features to their applications. One of the most popular products they offer is their SMS API. When sending SMS, it's really important to know whether or not a message was delivered. The Twilio API offers a "Status Callback URL" that will notify your app of delivery status after you send a message. Delivering a message takes a little bit of time so it's not possible to include a delivery status in the response to the original API call. Instead, you can pass a URL to be notified at asynchronously once the delivery attempt is completed.

To test this scenario, we can create a test that sends an SMS message via an outbound API call, then waits for a callback request containing the delivery status using the new Incoming Request Step type:

Incoming Request Steps can be validated just like the responses of outgoing API calls. In this case, we're checking for a form POST parameter named MessageStatus to make sure the value is correct. You can also check JSON attributes, extract and store values to use in subsequent requests with variables, or write more advanced assertions with JavaScript and the Chai Assertion library.

Testing Conditional Webhooks

Clearbit is one of our favorite new services. We make heavy use of their Enrichment API for qualifying leads. When looking up an email address, sometimes you get an immediate response (if it has been looked up recently) but other times you'll get a callback once the data has been retrieved. Using incoming requests combined with a condition, we can test this scenario as well:

If our initial request returns a 200 OK, we can assert on the response and extract data for subsequent requests. If we get a 202 Accepted back, we know that we'll get the details via the webhook URL and can handle the data there instead.

These are just a couple examples of how you can use Incoming Request steps to verify your webhook and callback workflows. There are many other possibilities including testing bot responses, simple API integration and workflow automation, and much more.

Now Available for All Runscope Accounts

Incoming Request steps are now available for all free and paid Runscope accounts. If you're new to Runscope, be sure to sign up for your free trial to try it out. You'll see the "Add Incoming Step" option at the bottom of the test steps editor. As always, if you have any questions or need help getting started, contact our support team any time.


Categories: announcements, monitoring, testing

Connect Runscope with xMatters for Automated, Intelligent Communication About API Incidents

By Ashley Waxman on .

When your business runs on mission-critical APIs, it’s important to know when those services are experiencing problems such as degraded performance, data quality, or simply not responding at all. Also, as more and more APIs proliferate to other parts of your application and IT infrastructure, knowing exactly who to notify about API problems depending on when and where they occur becomes even a greater challenge.

Intelligent Communication for API Incidents 

Today we’re announcing a new integration between xMatters and Runscope to combine the power of xMatters’ intelligent communication platform with Runscope’s automated API testing and monitoring solution.

xMatters connects insights from any system to the people who matter in order to accelerate essential business processes. The extensible platform allows you to add a layer of intelligent communication across the entire enterprise with automated, targeted communications ensuring that critical messages and notifications get through to the right people on the job. Runscope monitors APIs for availability and correctness. Combined together with this new integration, service issues detected by Runscope are routed to and handled by your existing xMatters’ communication plan.

Integration in a Snap

Connecting Runscope to xMatters is easy with the new xMatters Integration Platform. Once the xMatters has been configured to accept Runscope API test result webhooks, you just need to enable the webhook notifications in one or more of your Runscope test notifications. The Integration Guide has complete setup details.

With the xMatters integration, notifications about test results can be sent to groups or individuals. Team members can respond with “Rerun”, which will trigger another test run in Runscope and ends the event within xMatters. xMatters can also create FYI events to notify groups or individuals when Runscope detects a Live Traffic Alert. The notification includes a link the recipient can click to view the details and acknowledge the notification.

Let the Right Teams Know About API Issues Today

Runscope customers can begin integrating Runscope monitors with xMatters today. If you aren’t already monitoring your mission-critical APIs, sign up for Runscope for free today and see how you can solve API problems faster with API monitoring connected with xMatters. Learn more about the integration on the xMatters blog and check out the full announcement.

Categories: announcements, howto, integrations, testing, monitoring, product

3 Things Nobody Told You About API Development

By Ashley Waxman on .

Many factors in API development are brought to light after development is done: documentation, testing and caching. However, in order to develop APIs successfully and have an easier time maintaining and iterating on those 3 things, it is critical to include them in the development stage, and not as an afterthought. We recently sat down with Phil Sturgeon, author of Build APIs You Won't Hate, to debunk some of the misconceptions around what to include in development vs. post-production. 

In the video below, Phil explains how and why to include building tests, documentation and caching during the build phase of the API lifecycle. This video covers: 

  • A pragmatic approach to ensuring that your API solves real problems
  • Methods and tools for including documentation, testing and caching in the API development process
  • Basic principles of API design

You can start including API testing in your development process today by signing up for Runscope for free. Easily import your tests from the work you've done in Swagger and experience better test quality and coverage. You can also stay in the loop on discussions around API development by joining Phil's Slack channel

Categories: community, testing, webinar, api ecosystem

Everything is going to be 200 OK®