Sometimes we will need to use a service like Cloud Bigtable to store a bunch of data for our application. It sounds like an awesome option, with many built-in capabilities and incredible scalability. All sounds great, just one tiny question, how do I test my application?

In this tutorial, we will learn about some tools very useful to test the parts of our application that connect to Bigtable.

We will use a Bigtable emulator provided by Cloud SDK. Our application will connect to this emulator during testing, instead of the real Cloud Bigtable, this way we will not populate the real tables with test data.

It is important to note that the emulator stores data only in memory, which means that the data saved to the emulator will not persist across runs.

You can use the emulator with all Cloud Bigtable client libraries except for the PHP client library.

The emulator does not provide administrative APIs to create or manage instances and clusters. After the emulator starts up, you can connect to it using any project and instance name to create tables and read or write data.

Before we start, you need to:

gcloud components update beta
gcloud components install beta cbt

Running the emulator locally

Now, the next thing we will do is run the emulator locally, to get a sense of how it works before incorporating it into our project.

To run the emulator with default host (localhost) and port (8086), run:

gcloud beta emulators bigtable start

If you do not have the bigtable emulator installed, it will prompt you to install it, just run the command again when you are done.

And to set specific host and port (which we will, once we incorporate it to our application), run:

gcloud beta emulators bigtable start --host-port=[HOST]:[PORT]

Connect to the emulator locally

Next, we will want to connect to the emulator locally. If the emulator is not running, we can use the cbt tool to see what the real bigtable has in it, but how do we get cbt to connect to the emulator instead?

The answer is very simple, just set the correct environment variable and you are ready to go:

export BIGTABLE_EMULATOR_HOST=[HOST]:[PORT]

Remember the host and port you set when you run the emulator when setting the environment variable, in case that you used the default configuration, HOST will be localhost and PORT will be 8086.

cbt will automatically know that you want it to connect to the emulator, instead of the real Cloud Bigtable. It is important to remember that while this variable is set, cbt will connect to the emulator, that is why you need to unset it when you are done, you can do this with the following command:

unset BIGTABLE_EMULATOR_HOST

Now that the emulator is running and cbt will send the commands to it, we can execute some sample commands. To create a new table:

cbt -project foo -instance bar createtable foobar

You can indicate the project (in the emulator can be anything, I recommend using a random name, so you know you are connected to the emulator) with the -project flag, in this case, our project is named foo. To indicate the instance, use the -instance flag, in this case, our instance’s name is bar. Finally, we are indicating that we want to create a new table in the instance inside our project using the createtable keyword, the name of our new table, in this case, will be foobar.

When you run the command above, you will get something like this:

2017/06/14 16:24:15 -creds flag unset, will use gcloud credential

Another command we can run is:

cbt -project foo -instance bar ls

This command will show us the tables we have in the project and instance indicated, in this case, we will get:

2017/06/14 16:24:17 -creds flag unset, will use gcloud credential
foobar

We obtain foobar, the table previously created. To consult more commands you can run using the cbt tool, visit this site.

Incorporating the Bigtable emulator to your project

In this section of the tutorial, we will talk about a way we can incorporate the Bigtable emulator to our project.

First of all, let’s talk about some considerations. It would be easy to assume that if we have the emulator running locally and the environment variable set, this would be enough for our application to connect to the emulator during testing, and in some cases, that assumption is fair and right.

If you are using a client library to connect to Cloud Bigtable provided by Google (or by the community with a fair amount of support) and you are running the application locally, then it is more than fair to assume that your application will, in fact, connect to the emulator with just this local configurations. But this is not a very portable model, it requires some configuration and manual steps before you can run the tests that will connect to the emulator.

If you wrote (or are still writing) a client library, then you have to make sure that your library has a way to recognize the environment variable is set and connect to the emulator.

The following steps are to configure the Bigtable emulator in your project using docker-compose:

Step 1

The first step is to create a container that runs the emulator, for this, we can use the google/cloud-sdk docker container from Dockerhub. Add this to your docker-compose.yml file:

services:
    ...
    bigtable:
        command: gcloud beta emulators bigtable start --host-port=0.0.0.0:8086
        image: google/cloud-sdk:latest
        volumes:
            - .:/opt/folder_name/my_application:delegated
        working_dir: /opt/folder_name/my_application
        environment:
            - BIGTABLE_EMULATOR_HOST=bigtable:8086
        ports:
            - 8086:8086

As we can see, a command to start the emulator is executed when the container is started, and the ports we defined are the default mentioned earlier (8086). Now when we run:

docker-compose up bigtable

We will have a container with the emulator running, great!

Step 2

Then, we can add a service that runs our application when we want to run tests, and add our bigtable container as a service our application depends on:

services:
    my_application_tests:
        depends_on:
            - bigtable
        volumes:
            - .:/opt/folder_name/my_application:delegated
            - ~/.config/gcloud:/root/.config/gcloud
        working_dir: /opt/folder_name/my_application
        environment:
            - BIGTABLE_EMULATOR_HOST=bigtable:8086
    ...[other configurations needed]

In the example above, we are not adding all the other configurations needed to run our application in a container, because that depends on each application, so, when creating a container that runs your application for testing, do not forget to add all the configurations needed.

Now, when we launch our application container for testing, the bigtable container will also be started:

docker-compose up my_application_tests

The command above will start both the application’s container and the bigtable container, we will not nned to run docker-compose up bigtable to have the emulator running.

Step 3

Now we have two containers running, one with our application and one with the Bigtable emulator, the next thing we want to do is run the tests, now that the environment variable is set in both bigtable and my_application_tests containers, we do not need to worry to set it anywhere else. We will use a bash file to start the containers and run the tests.

I also recommend having a bash file that specifies the specific commands needed to run the tests in our application (we will not include an example of this file, because it is highly reliant of the language your application is written in).

Following is an example of a bash file that starts the containers and run the tests (by calling another bash file that contains the exact commands needed to start the tests execution):

# run_test_bash

echo "=> Starting containers"
docker-compose up -d my_application_tests

echo "=> run tests for our application"
docker-compose exec my_application_tests ./test_bash

In this bash, we are assuming a bash file that specifies commands to run tests already exists, we are also assuming that it is in the same folder that this other bash file (run_test_bash) is and we called it test_bash

Step 4

Let’s run our tests by running the bash above (we are calling it run_test_bash in here) and wait for the results:

./run_test_bash

Assuming the bash run_test_bash is in the folder we are currently in when running these commands.

And we are done.