All Blog Posts

1 post tagged with "cloud"

View All Tags

Fast autogenerated code deployment in Google Cloud Functions using Node.js

Santiago Arranz Olmos

Imagine Compilers Team

Background

At Imagine we wanted to provide users with an all-in-one backend development experience while using the tool. From our web app, you can start a project from scratch and quickly have a backend live to play with. We wanted our users to be able to interact with their project seamlessly, and also perform tasks they usually would do locally directly from the web app, such as running tests, linting the code base, and interacting with the database.

The challenge of this task is to be able to deploy multiple entire backends quickly, allowing each user to interact with the API and directly with the database, all while running tasks on the code (linting and testing). Moreover, since with Imagine you can generate code in different frameworks (Django and Express) and with different API formats (REST and GraphQL), this environment should be available and consistent regardless of the framework-API format configuration.

Deploying multiple backends on-the-fly

We use a Node server to handle all interactions between users and their backends. It is an unusual undertaking for a server because it entails routing requests to their respective servers, and fetching and deploying said servers on-demand. Thanks to Javascript's and Node's versatility, we were able to handle all this in a very small amount of code, having our server lazily deploy a user backend and transfer control to it when a request is received.

/* Get the user project and forward the request. */
const main = async (req, res) => {
...
const app = await getApp(projectId);
app.use(cors())
await app.handle({ ...req, url: `/${projectUrl}` }, res);
...
}
/* Load user project if possible. Otherwise, deploy and load. */
const getApp = async (projectId) => {
const pathProjectDir = path.join(PATH_PROJECTS, projectId);
const pathProjectApp = path.join(pathProjectDir, 'dist', 'server', 'app.js');
try {
chdir(pathProjectDir);
return require(pathProjectApp).app;
} catch (e) {
await deployProject(pathProjectDir, `${projectId}.zip`, projectId);
chdir(pathProjectDir);
return require(pathProjectApp).app;
}
}

Interacting with the API and directly with the database

Django has an excellent /admin page where you can interact directly with the ORM, and countless developers find this an excellent feature. We wanted to replicate this across all supported frameworks and API formats. After researching alternatives for admin panels, we concluded that the best experience is achieved through the default admin panel in Django and AdminBro in Node.

Similarly, an important instrument all backend developers need is an effortless way of using the API. For REST a well-known toolset is Swagger, which is the best alternative according to our research. For GraphQL the answer is straightforward: GraphiQL, which is supported by the GraphQL organization, provides everything you need in this respect.

Testing and linting generated code

Imagine-generated code ships with auto-generated tests and you can watch the tests running and check the coverage report right from the web app. We implemented this using a Cloud Function, which streams the test output live to the web app through Redis and uploads the coverage report afterward for the web app to see. Linting is similar, although no live streaming is necessary. As we want our users to interact smoothly with the generated code, it is imperative that the generated code be displayed as soon as possible. The core compiler is extremely fast, being compiled and optimized, but linting the files relies on external tools that take time. We therefore compromised and display code that has not been linted (though we are working on making it tidier!) but the linted code is available for download after a couple of seconds.

Conclusion

The Imagine compiler and its web app together comprise an orchestra of components that come together to deliver the best backend development and deployment experience possible. Implementing these building blocks required pushing available tools to the limit, and presented all kinds of challenges.