Even if you are an initiate to software development, chances are you’ve worked with an Application Programming Interface (API). For those outside of the tech world, many are using and/or benefitting from an API whether they realize it or not. Indeed, the interoperability of everything around us is achieved in large part due to an API someone made, and its ability to communicate with other systems. Just like anything in software development, creating an API for someone to consume can be challenging. You run many kinds of risks, such as creating something counterintuitive or difficult to use.
This is part two of a three part blog series focusing on REST API design, and sharing some advice and mistakes I’ve made over the years while designing APIs. If you haven’t read the first post yet, check it out here. In this post, I will focus on an important aspect of API design – handling and adapting to your business domain changes through versioning.
When you deploy your API, consumers will begin using it for the currently released version. When you go back and make changes, however, you can be faced with a very big problem. Let’s consider this scenario: You create a baseball API that tells you scores (I’m making this absurdly simple for our example). Let’s say you create an endpoint like this to get the score for the latest (or current) New York Yankees game /api/scores/mlb/teams?name=Yankees. This returns a JSON response like this:
So clearly this is very basic. It doesn’t do much, but it does what its supposed to do. Your simplicity is attractive, and a lot of other websites decide to start using your API for their score aggregators. Let’s pretend a few .NET projects decided to incorporate it, and then finally a mobile developer created an iOS app to fetch the score from your API. Great! After a few months, you decide it’s time to improve upon the API. Specifically, you received requests to see more weather conditions on the field. This will be applicable to current games (or games that are about to start) so someone can figure out if they need to bring an umbrella or not.
You figure with the right type of response, someone who uses your API could do something like this:
You also didn’t like that “score” was a string, and you realized the name “currentScore” was a bad choice for past games. So you change all this. After making the changes and pushing the latest code to production, you test your endpoint and your response returns this instead:
So this looks good, and you pat yourself on the back at a smooth code deployment. After a week, you start to get angry e-mails. “You broke the API!”, “It doesn’t work anymore!”, “Thanks for not telling me you changed the JSON!”. Whoops! You forgot that in changing the API, people created their own mashups featuring your original design, and now they have to update their production code to match yours. If you ever did a prod deployment, you know sometimes that’s not that easy to do. Not cool. But if you revert your code back, you’ll break the new consumers of your API! The horror!
The above scenario was very simple and in the real world it’s often not as egregious as that. I assume it’s pretty obvious to most that a change like that creates problems, however what about hundreds of endpoints and very complex JSON documents that are returned? Small changes like a data type from a string representation of an integer, to an actual integer (“2” vs. 2) could be difficult to pick up on. Large changes, like completely redoing an entity or updating the cardinality of database tables, can force you to redo an entire controller. This is going to happen, and you will need to be prepared to keep things running smoothly.
How could you fix the above API response to satisfy both the original users, and the new ones? Well unless you could force them all to update (forced updates happen a lot in the mobile community), I will explore some of the techniques available.
Oftentimes you had to make changes to your API because you added new functionality, and with new functionality comes new properties. If you find yourself in this situation, where the original representation of the entity hasn’t changed, then you are in luck. If the original JSON representation of an entity had 20 fields, and you add 5 new ones, then your old API consumers will not be affected, and your new ones can simply take advantage of the new JSON response. This is due to the fact the original JSON hasn’t been removed or renamed. When a consumer is ready, they can update their code to take advantage of your new fields, but there is no rush. Essentially you’re creating a response in such a way where everyone wins. The old consumers have their contract, the new ones have theirs, and both are happily data binding.
Unfortunately, while one could get away with some JSON responses simply having new properties, the reality is that you will eventually get into a situation where you need two different JSON responses on the same route. So how do we do this?
Versioning is the idea that as your data models undergo different versions, so too does your API, and subsequently, the response that comes from the API. Just to be clear the response returned is based upon the version requested by the client.
There are three common ways to do API versioning. I’ve seen other approaches, but these are the three I think are worth mentioning. Everyone will tell you your versioning strategy is flawed in some way. You’ll have to have some thick skin if you’re working on a team or with some tech leads.
- Query String – No surprise, the client uses a query string to identify version of a resource it is asking for. So as an example, /api/v2/scores/mlb/teams?name=Yankees is version 2 (v2) of the API (the above updates we made) whereas /v1 is the first version. I’ve also seen folks do it like this /api/mbl/teams?version=2&name=Yankees (so an actual query string parameter). Typically you would have original URL map to v1, and you manually specify /v2/. This means /api/scores/mbl is equivalent to v1, and /api/v2/ is obviously v2.
- Custom Header – Remember those X- headers I discussed in the previous blog post? You could certainly pass an X-Version: 2 header or something similar, and the API can base the response off this.
- Accept Header – This is actually using an existing header – Accept – to get back the correct version of your API. An example would be Accept: application/vnd.scores.api+json;version=2. The ‘vnd’ part gets used to differentiate custom media types such as what we’re passing in.
…What They Don’t Tell You
If you’ve looked around most blogs about API versioning, you rarely see them go beyond the above subject matter. That is to say, you’ll get some great strategies about how to handle versioning in your URL path and how to hit your API accordingly. What you don’t see, however, is the messy underlying details to make the above happen. In fairness, the reason for this is because it’s a difficult thing to get into exhaustively to be of some use to someone; everyone has a specific architecture and set of problems to solve. As a result, you’re not going to find too much more in this blog post but I’d at least like to take a stab at addressing the high level issues. It’s not easy.
My recommendation is to assume you will need to change your API every few months, and that you will need versioning. Keeping this mind set will force you to write code in a manner where you prepare for versioning in advance. When you are in the early stages of development, so much is going to change that it probably isn’t worth actually doing the versioning, however you should always consider it and design your code in such a way so that when you go live, you are prepared to version. Let’s now go into some details.
Interpreting a Version
In all of the above examples, you must write code that interprets the version requested serves a different response. First, you need to decide on your versioning strategy. My advice is to have a default (unspecified) version always be the first version, this way you don’t break the existing users of your API, and then you can inform others about your new version (who can access it using v2). With analytics, you’ll be able to tell if everyone has eventually moved over to your new version (although this rarely happens in any sort of short timeframe).
You should also decide on how long you want to support an older version before making it the v2. I’ve received e-mails from certain groups mentioning that they are upgrading their API in 3 months and to start using the new version to prepare for the cut over. Twitter and websites are also good ways to communicate this. Here is an example from Github about their messaging and strategy. Of course, some people never change it because they want to support everyone. That is also an option. You need to consider what works best for you and your consumers.
Now that you decided on how to handle the whole v1/v2 thing, it’s time to decide on how “versioned” you need to be. There are two strategies I’ve seen. I like the first one better, but have to recognize the second one as an option for some folks.
- The same codebase will support multiple versions. Have your web server interpret the IIS requests and route traffic to the appropriate endpoint. You can do this with an htaccess file or url redirects. Once you do this, your code is then responsible for handling the versioning accordingly. We’ll get into that more down below.
- Multiple codebases support one version. This can be done by cloning your current codebase, and setting up a completely new site. Your web server will route traffic to the site based on the URL (you can map a cloned physical directory to your request via your web server). While this sounds crazy, some APIs are simple enough or designed in a way where this could work. I personally don’t see this method too often in the enterprise. If you’re cloning your codebase you need to handle the question of what to do about your database. If you share databases and you’re using an ORM then you will have mapping problems that will need to be addressed. If you use different version of databases or tables, then you need to handle data reconciliation later (think about if users registered, for example). If you have a super simple API going on, however, something like this might be easier for you to do. You can consider some variation of this multiple codebase approach as an option depending on your needs.
Data Transfer Objects (DTO)
Data Transfer Objects are flat representations of entities. Traditionally they are just a bag to hold all of the fields of an entity, and do not contain business logic, methods, computed fields, or anything else. Architects would say this is an anemic entity. I’ve seen some circles begin to say they should do more than that, so I guess at the end of the day just do what makes sense for you. If you need something special in all your DTOs, then go for it.
A good practice is to take your entity and represent it as a Data Transfer Object before returning to the consumer. In doing so, you avoid exposing the entity directly to the consumer. This means instead of the fields on the entity itself being converted to a JSON response, the fields on the DTO are returned as a JSON response. What’s the difference? if you have 10 fields on an entity, and the same exact 10 fields on a DTO, then aren’t you returning the exact same thing? A few points to address this before we move on:
- Doing this comes with a cost. You’ll definitely spend more time wiring this all up and maintaining it. There are also tools like AutoMapper that assist in mapping fields. I like to use this as it also allows you to write custom code for specialized fields.
- When something changes (versioning needs to happen) your entity is directly returned. This is generally not a good idea. Sometimes we want more fields, or less fields, than what’s on the entity. Sometimes we don’t even want a full representation of your entity. For example in many GET requests, having certain fields returned is extraneous. I always find it more efficient to return the bare minimum.
- Similarly for PUT/POST, you don’t always need the full entity. Let’s consider a user registration form – if your user entity has 20 fields on it, but your registration only contains 5 fields, then it’s better to have someone send over a UserRegistrationDto which just has the 5 fields, then make them use a 20 field DTO. Default values in this example would perhaps mitigate this, but what about in the case of a PUT when you update the entity? If a particular form for updating your communication settings exists, and it only has 3 fields on it, then technically sending over the entire User entity would invalidate the other 17 fields (making them null or default). Instead, having a 3 field communication DTO can leave the other fields intact, and only update the 3 relevant ones.
- Less traffic – simply put, if you have smaller bits of data going over the wire, it’s just more efficient. I understand this could be milliseconds, nanoseconds, or non-existent, but the fact is having much less go over is better. I’ve seen full entity graphs for something like a user entity have 150 fields on it with all related entities. Sending 5 fields is just better.
- When you need to transfer an entity from different areas of your application, and ultimately to the consumer, it’s easier to have a flatter version of your data to ferry around without having to refetch the entity from the database every time. This can save time.
Underlying Code Changes
Let’s assume you’ll go with approach #1 above, and that you routed your version to the controller. It’s now time for your controller to interpret this versioned request and return the appropriate response. Where you go from here will really depend on the programming language and the code you wrote. If you completely changed your database table as a result of your next version, then your underlying entity is going to look different. Assuming you can’t just return the whole entity like I mentioned earlier, then the root of the problem is that some consumers want the previous representation of an entity, and others want the full representation. If you have three versions, then the first two want a subset, and the third client wants a full entity.
Data Access Layer
If you need two completely different SQL statements to run for getting back DB results, create a new method in your data access layer to perform this fetch. You will then use this result in your service layer, controller, or wherever you need to access it. In many cases, this isn’t good enough or runs the risk of code duplication. If you need the same entity returned back to you then return the entire updated entity back up the chain.
Obviously your code architecture is going to vary here; some architects want to access the data from their controller via a repository, some will have their controller call a service to call a repository, etc. A good strategy is to have your data access layer return the entire updated entity back up to a service class . This class has the responsibility to then analyze the current version requested. Then you can make use of data transfer objects (DTOs) to represent different versions of the entity, and return each one, according to the version requested, to the next layer up based on your architecture, and ultimately to the client requesting the entity.
On the most basic level, you could have some if..else statements in a service layer that returns the representation of the entity that is contextually relevant for its version. I think this runs the risk to be messy if you make frequent changes, and the class can grow to something massive. On the other hand, writing an extremely elegant framework might not be in your timeline either. A good stake in the ground would be to get as much done as you can in some sort of class, and then plan to refactor into a framework.
As far as versioning goes, your controllers interact with a repository or service layer to get back your DTOs. I think everyone has an opinion on the “right” way to do this – this will be beyond the scope of this post. If you can get back the DTO-representation of your entity, then that’s good enough. One thing I would avoid is putting intelligence directly into the controller to handle versioning; you want your controller to have one responsibility – to handle the resource operation and response. Everything else will get delegated via action filters, delegating handlers, service layers, and custom classes. Handling conversion of an entity to a DTO, deciding which version to pull, and assembling the right JSON should not happen within your controller methods directly.
In this post, I covered an important aspect of API design – handling the inevitable of your domain model changing, and how your API responds to this. Versioning is an important approach to making all of your consumers happy. In the next and last post, I’ll cover selecting, sorting, and pagination of resources with ODATA, as well as tools to use for making API calls and more. Stay tuned for it to be finished!