Make Your Python Project Readable and Sustainable From The Beginning
Lessons learned from a real example.
Recently, I work on a gigantic web application written in Python. It makes me feel challenging since I was not involved in a Python project for a long time. It is helping me not only to remember the most of the practices that I used before, but also to learn the ones that I have never been aware of. After noticing that I tend to forget easily, I decided to keep organized the experience which is worth noting it down. I will try to share some of the challenges and the way how I dealt with them, both to give people — facing with similar problems — a different point of view and to record them for referring in the future.
In my humble opinion, one of the most essential tasks — not for the sake of the project, but for keeping the developers’ sanity in a reasonable level — assigned to me in the project so far was about configuring tests, formatters, linters and incorporating into new CI/CD pipeline.
Before going deeper on the task, it would be appropriate to elaborate what we have as an application.
Application
The application itself has eleven different services written in Python. They communicate each other via an HTTP Rest API, websocket or message brokers. All these services are being developed for about 5–6 years by unknown number of developers.
Complaints
As we all know, each developer has his/her own style. As long as programming language let you do something, developers feel themselves free about doing it. At least this is what I observed so far. After a quick reading on the code, the parts written by someone else can be identified easily.
It is really no big deal having different styles in the code base as long as it runs as expected. Especially, if you are a contractor, you will be done when your contract expire. However, trust me, your efficiency on the project will not match with a project that you feel comfortable.
The major problem is tests are not functional. A huge number of integration tests were written along with a few unit tests. However, either most of them became obsolete, or disabled due to bring in some features breaking those tests. After a short period, nobody ran tests even locally.
Imagine, a huge amount of money flowing through your application and you will never know that you have a bug transferring all of your money to a bank account in Caribbeans before it really happens.
Isolation and Managing Dependencies — poetry
Two years ago, when decided to quit my daily job that I was coding Python everyday, we were using bare virtual environments along with pip. I don’t really remember that if we needed anything which is capable of more than we had. To be honest, I didn’t really know that if there was such a tool.
In the current application, we are using pipenv. Although, I am not so familiar with it, I don’t think that it gives a smooth development experience. My first impression is that it looks like arbitrarily can/cannot install some of the dependencies. I know that I am not alone about this, since we have a task to migrate to poetry in the backlog.
After a quick search on poetry, I decided to use it on my new projects (Also, someday we will migrate in the application as well). So, created a project with poetry. For the details of poetry, please visit its documentation.
Tests
Poetry, creates a test directory and automatically adds pytest into your project dependencies.
Writing tests for your application from the beginning is way much easier than writing tests for a complete application. So, it is so nice to have tests at step zero.
Another thing is that how will you decide that you will write enough tests? At this point, coverage tools can be helpful. I did not add the application that I created since I don’t have any test at all.
Readability — Linters, Formatters
To keep your application readable and maintainable it is essential to have a standard way of coding. Linters and formatters serves as to keep your code base in an ordered way and does not let anyone to destroy this decency if integrated well (e.g. in CI/CD).
I preferred to use black, isort, bandit, pylint. The configurations that I used can be found in github repository.
Automation Scripts — usage for CI/CD
Poetry lets you write build scripts just as npm did in the package.json.
In pyproject.toml :
[tool.poetry.scripts]
linters = "bin.run_linters:run"
formatters = "bin.run_formatters:format_all"
check-formatters = "bin.run_formatters:check_all"Notice that, under bin package, we have run_linters.py and inside it we have run() method. That’s it.
Further on maintainability and sustainability
From my previous experience, one of the painful things is type checking. As a precaution, static type checking can be added for further enhancements.
Also, coverage tools would be handy and useful.
Conclusion
Based on my recent experience and research, I tried to share my findings on how to keep a Python application readable from the beginning.
The github repository that I plan to use as a base.
