Spring Boot ChatGPT Microservice

TLDR; 

In this blog I’ll explain my journey to create a Spring Boot Microservice which, given an input from a web client (Postman, a UI frontend), it returns the response from ChatGPT (the example uses the gpt-3.5-turbo model). 

If you’d rather skip to the code, you can find the GitHub repository here. 

Sharing the journey – lessons learnt

The first lesson I’d like to share has to do with the development lifecycle when using a third party API. 

The development lifecycle when using a third party API

The assumption is that the API provider has well-written documentation and, most importantly, good JSON examples for both requests and responses. Luckily, the Open AI API has got these. One approach that worked well for me was the following lifecycle:

• Save the example JSON request/response files locally to your IDE, under the test resources of your code. This will allow you to write unit tests without having to write a complete integration with the API provider. 
• Start by writing a unit test to validate the conversion of JSON files to business objects. Of course as you’re reading this blog you might be asking yourself: which business objects? I used TDD to produce the Microservice described in this article. By writing the tests to validate production code that doesn’t yet exist, the business object (and everything else) will naturally emerge by way of making your test pass. 
• For the JSON to Business Object and Business Object to JSON conversions, I strongly recommend the Jackson library, which comes natively with Spring Boot as well as the Lombok library to avoid much of the boilerplate code in the business objects, like constructors, getters, setters, equals and hashcode. 
• Organise your product (not test) code with the following structure:
  • main: this should contain only the main app or any low-level code. As explained in one of my articles, the main partition should depend on the app partition
  • app: This should contain the core of your application, including configuration, models and services
  • view: This should contain the view code (e.g. Controllers) and should depend on the app partition

All partitions should depend on the app partition. 

At this point I feel I should clarify something. According to the article mentioned above, the app partition should only contain interfaces, with implementations residing in the main partition, as main (as well as view) should be a plugin to the app. However in this particular example, since there is only one service to interact with the Open AI chat endpoint, the OpenAiChatService resides in the app partition and there isn’t an interface for this service, since there is a single implementation. The Service is the implementation. This does not break source code dependency management as the main partition contains only the bootstrap code. All the dependencies in this application point towards the app partition and the app partition doesn’t depend on any other partitions. This enables independent deployability thanks to the fact that source code dependencies all flow towards the app partition and the app partition doesn’t have any source code dependencies towards other partitions. 

Continuing with the development flow: 

• Place the business objects in a model package within the app partition. For example, for each JSON example downloaded from the API provider’s website, I’ve created an equivalent business object in the model package. By using Jackson and Lombok annotations, the resulting code is clean and easy: 

The @Data, @NoArgsConstructor and @AllArgsConstructor Lombok annotations provide the byte code with constructors, equals and hashcode as well as getters and setters. 

The @JsonProperty annotation is a Jackson annotation and its value must correspond to the key name in the JSON file. This allows to name the properties in the Java code whatever we like. Jackson will automatically read the JSON file and map its values to the corresponding Java properties. Please note that if the JSON and Java property names are the same, you don’t need to annotate the Java property with @JsonProperty. 

I’d also strongly advise the usage of Collections rather than arrays. They both work equally well when it comes to serialisation / deserialisation but Collections are more developer’s friendly. 

An example of a unit test for JSON to Business Object mapping using Jackson follows: 

If we look at the first unit test this is pretty easy: 

• It manufactures a Business Object to hold the equivalent of the JSON payload required to make a request to the Open AI API
• It reads the locally saved json payload into the same Business Object type, using Jackson’s ObjectMapper
• It verifies that the object created from the JSON file is equivalent to the object manufactured by the utility class. This tells us that the conversion from a JSON file (which will be the payload returned by the API when we call it for real) works correctly

The integration lifecycle

Once the JSON to Java (and viceversa) mapping has been unit tested for all payload involved in the third party API interaction, you have a solid basis for the next step, which is to write the actual integration production code.

While writing the integration code it’s useful to consider the following: 

• The application must not expose any sensitive data (like your API key). It’s very likely that you will manage your code through a Source Code Management tool like GitHub. Don’t commit any sensitive data to your repository because, once committed, it becomes public information. The best way to avoid this security concern is to apply this principle before you start writing any integration code (in fact any code). Thankfully, Spring Boot is really good in its support for Cloud native applications, that are written according to the 12 Factor apps principles. One way to enable your application to use sensitive data while not exposing it, it’s through the use of environment variables. For example, I’ve setup the value of my OPEN AI API KEY as an environment variable and then used a valueless notation in my code to retrieve its value. This of course means that any environment where you will run your application will need this environment variable set. I used this approach also with GitHub, while using GitHub actions to automatically execute the unit and integration tests after every push. This approach can be seen in the OpenAiConfiguration class, which contains the Spring Boot programmatic configuration for this app: 

The ${OPEN_AI_KEY} notation instruct Spring Boot to search for the value of that property in the environment. Since the environment is private to my computer, I can safely commit this code without compromising the value of my API key. Later we will see how it’s possible to set secrets in GitHub so that these are accessible from GitHub Actions. In the code above I also create a HttpHeaders Bean because with every request to the Open AI API, these headers must be sent, including the API Key. 

Continuing with the integration development cycle: 

• Following TDD, start writing an integration test so that relevant Beans become available to your tests by simply injecting them. In this case, we want to make sure of the following:
  • The main application is annotated with @SpringBootApplication and we specify the base packages which the application should scan to find Beans. 

This is all there is to the main app. Please note that by explicitly specifying the app and view partitions (the last part of the package name) we’re enforcing good dependency management by saying that the main app depends on the app partition (and the view when you have controllers). 

• We want a configuration class in an app/configuration package to declare all the Beans that the application will use. This, again enforces the principle that all source code dependencies should point towards the app partition. In the Configuration class (annotated with @Configuration) we declare the beans that provide useful components for the Microservice we are writing: 

The API key and the HttpHeaders Beans have already been explained above. The only other thing we declare is a Spring RestTemplate which is the gateway to make HTTP requests. 

• We then create a Service (annotated with @Service) to handle the communications with the Open AI API. As this is the main function of the app partition, we want the Service contract to use Business Objects both for parameters and return values. The idea is for the Service to accept a Business Object with the content for making the request and return a Business Object containing the reply from ChatGPT for the response. 

The Service class is quite simple. First thing to note is that we inject into this class the endpoint to the Open AI Chat endpoint (this has been set in the application.properties file of the Spring Boot app as openai.chat.url). We can do this here because the URL is not sensitive data, therefore there’s no need to set it as an environment variable and keep it secret. The Service is then injected with the RestTemplate we declared in the Configuration class and with the HttpHeaders bean again defined in the Configuration class. Following Clean Code recommendations, we also have the Service declare its own exception as a subclass of RuntimeException. This is because we want exceptions to be as close to the scope as what triggers them as possible and with a name that describes what they are for. 

Of note is the signature of the makeChatRequest method. As explained earlier, it accepts a Business Object as input that represents the request and returns a Business Object that represents the Open AI Chat API response. This makes the api intuitive to use. 

• Define the Controller which captures the request from an HTTP client and uses the Service to get the ChatGPT response. The Controller is really simple: 

The Controller is annotated with @RestController, a convenience annotation that assumes @ResponseBody for the return type in the Controller methods. Here it’s important to note that the Controller is part of the view partition of the application and it depends on the app partition. This can be seen as the chat method accepts a Business Object defined in the app/model package and it uses the Service declared in the app/services package. There are no source code dependencies on the view partition. Jackson automatically maps the JSON payload in the request to the Business Object containing the request. We know this will work as we have written unit tests for it. It then invokes the Service method and returns the Business Object containing the response. The only other thing that does is capture any exceptions coming back from ChatGPT and wraps them with a more meaning status code / message to the client. 

Testing the application

To test the application, you can run the Spring Boot app. This will start the Netty Servlet container listening on port 8080. Then, you can use Postman to test a POST request to the url: https://localhost:8080/chat with the following JSON payload: 

{
  "model": "gpt-3.5-turbo",
  "messages": [{"role": "user", "content": "List BIAN domains for banking accounts and their main responsibilities"}]
}

The response looks as follows: 

{
    "id": "chatcmpl-74mgzxzscBmXQYD5rJIzwssWCKUQk",
    "object": "chat.completion",
    "created": 1681374841,
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "1. Customer Management: This domain is responsible for managing a bank’s customer accounts, including opening and closing accounts, maintaining personal information, and facilitating transactions.\n\n2. Financial Accounting: This domain is responsible for managing a bank’s financial accounting processes, including reconciling accounts, managing financial statements, and ensuring regulatory compliance.\n\n3. Credit Management: This domain is responsible for assessing the creditworthiness of potential borrowers and managing the loans and credit lines of existing borrowers.\n\n4. Payments and Collections: This domain is responsible for managing a bank’s payment and collection processes, including processing payments, managing collection efforts, and initiating automatic transfers.\n\n5. Risk Management: This domain is responsible for identifying and mitigating various types of financial risk within a bank, such as credit, market, liquidity, or operational risks.\n\n6. Fraud Management: This domain is responsible for detecting and preventing fraudulent activities within a bank, including transaction monitoring, customer profiling, and identifying potential fraudsters.\n\n7. Wealth Management: This domain is responsible for providing investment and financial planning services to high-net-worth clients, including wealth protection, wealth accumulation, and wealth distribution.\n\n8. Regulatory Compliance: This domain is responsible for ensuring that a bank complies with all applicable regulations and provides timely compliance training to its employees."
            },
            "finish_reason": "stop"
        }
    ],
    "usage": {
        "prompt_tokens": 19,
        "completion_tokens": 255,
        "total_tokens": 274
    }
}

Look at the value of the content attribute in the response. The value of this property has been returned by ChatGPT, using the model gpt-3.5-turbo.

Continuous Integration with GitHub Actions

I remember when to run by CI/CD pipeline I had to have an instance of Jenkins (locally or on the Cloud) with all the maintaining that came with it. Gone are those days with GitHub Actions. They allow you to get GitHub to run your build upon push. They support different build tools (e.g. Maven and Gradle for the Java enthusiasts) and several options: e.g. you can decided to launch a build upon push on all your branches or select the ones you want. 

GutHub actions enable you to also build your Docker images and ship them to your container platform of choice. 

For this particular example application, I use them to run the Maven build, which also runs all unit and integration tests. This gives me the piece of mind that my code can be shipped if there are no test failures. 

Secrets

As discussed earlier, integration tests will need access sometimes to real sensitive data to run, e.g. your API key is you’re using a third party API, or your login credentials into a container or Cloud platform. Writing and committing these sensitive information in the code is a no-go from the start. 

GitHub allows you to define secrets. You can define them at the account/organisation/repository level. 

GitHub Actions allow you to have access to these secrets. Let’s see how it works. 

First, you want to go in the Settings (account, organisation, repository) where you want to make these secrets. In the case of this application, I have created a secret at the repository level by clicking on the Settings for my repository. 

You then click on Secrets and variables and choose Actions. From here you can select New Repository Secret and add the name of your secret and its value. GitHub stores this information securely. 

In the case of this application I have created the secret OPEN_AI_KEY and assigned to it the value of my Open Ai API key. 

Using GitHub Secrets within GitHub Actions

To enable GitHub actions for your code, you can create a .github/workflows folder and create a yml file in it. In my case I called it maven-build.yml. Here it is: 

When you push your code, GitHub will look for this file and it will run the corresponding action. Let’s look at some of the properties in this file: 

• on: This indicates on which GitHub user’s actions this action should activate. You can customise this property by also specifying the branches you want this to work for. 
• env: Here you can specify which environment properties should be made available to your build. Remember when earlier in the post I said I was using the environment property ${OPEN_AI_KEY} in the Spring Boot code to keep secrets safe? Well, here I define the same environment property. Please note the value that I assign to this property: ${{secrets.OPEN_AI_KEY}}. This is where the magic happens. When I push the code, GitHub Actions will run this action, it will retrieve the value of the secret we have set up earlier and it will assign it to the OPEN_AI_KEY environment variable. This is how my integration tests can run successfully. The Spring Boot runtime can resolve that environment variable.
• jobs: Here we define that we want to run a build job on the latest Ubuntu distribution. 
• steps: Here we define all the steps required for the build. Namely:
  • Checkout the code
  • Setup the Java JDK (version 17 Temurin in this case)
  • Run the Maven command to build and package the code. This will also run all unit and integration tests (indeed every test) in your code

Checking the results

Every time you check your code in the GitHub Action will run, invoking the Maven build and building your tests. To check the result, head to your repository, click on Actions and you will see the results. 

You can click on each build and see the logs of what happened. If the build fails, GitHub will notify you via email. Isn’t this cool or what?