Data Solutions Portfolio
DONNIE J.
​API Development with Python
Overview
Creating a fully developed and hosted python application which serves live data via custom API endpoints.
The Python application serves an interactive frontend dashboard that connects directly to custom internal API endpoints. When triggered, these endpoints query a live cloud database, transform the retrieved data, and pass it to JavaScript on the client side for real-time visualization.
Development
The development workflow for the live application involves many details and parts that can be categorized within the major steps in the following chart. I will go into brief detail on each stage for a better understanding.
Starting with the Environment Setup phase, it is important to define what packages and resources are required for the project. This includes how the repository needs to be set up and the less obvious resources, such as the best format for project credentials.
This also includes Python environments. Since this project will be deployed and depend on specific versions of packages, I created a separate virtual environment locally solely for the application development.
A major part of the setup phase is creating a cloud-hosted database to store the data. For this, I used Neon to host a Postgre18 database.
With the database setup, it is possible to do further backend development, such as testing connections and database querying via Python. To accomplish this, the Python Flask and SQLAlchemy frameworks are used.
With these two coupled packages, it is possible to create many components needed for the application using Python.
Once database connectivity is figured out through python I map the database object(s) to a python class. From there the custom endpoints and logic for transforming the data can be developed.
The frontend layout is created using HTML and CSS. The data is passed through to JavaScript functions, which load the libraries used to visualize it. Python transforms the data into the format expected by JavaScript. This allows all of the heavy lifting to be done by Python while the frontend focuses solely on displaying data.
​
Handling data logic with Python instead of inline JavaScript ensures better maintainability. It simplifies debugging and provides the flexibility needed for future data transformations.
In the final stages, in order for end users to access the app, it needs to be properly hosted for optimal availability and web traffic. For this, I am using Render. Within Render, the application code can be deployed and version-controlled, allowing for a CI/CD workflow where I test new features and troubleshoot locally before deploying new versions, with the constant option to rollback to a stable build. This means I can ensure application uptime for end users.
Challenges
Every Project has its hurdles to overcome. These were some of the challenges for this project: ​
-
Controlling Costs: Being a personal project, resources are limited, so tools such as cloud database hosting must be run at a lower service tier. This results in longer startup times and quicker connection timeouts.
-
Optimizing the Frontend: Improving user interactions and leveraging UX design elements and prompts to circumvent the "slow start" feel of the application.
​
Future Updates
Projects are pieces of work that can be continuously revisited and developed. Some future updates I would like to explore include: ​
-
Complex Transformations: Introduce API endpoints capable of multi-step data transformations and real-time aggregations.
-
Enhanced Visualizations: Expand frontend features with richer, interactive charts.
-
Parameterized Endpoints: Enable user-driven filtering by accepting custom query parameters at the endpoint level.