Tame (Py)cURL Part 2
Previously, we discussed the possibility to hack PycURL to achieve somewhat a more controllable HTTP streaming client.
Today, let’s add some Redis ingredients into our recipe and make this tool more controllable so that we can solve the remaining problems aforementioned.
The rational behind utilizing Redis is that we want a scalable (both server and client), and easy-to-use in memory key-value store to maximize our control over PycURL. With that in mind, I actually encouraged the company that I work for to open source this project I created during my day time job.
Introducing TideHunter:
- Accurate quota limit - total control over your stream quota.
- An instant off switch - when sht hits the fan and you don’t want to crash your process.
- Redis backed control tools - semi-persisted, fast, and scalable.
- Core mechanisms based on the solid cURL and PycURL - inherits the built-in goodness (gzip support and more).
- OAuth support based on python-oauth2 - see this demo in action.
This project is submitted to PyPI, which means that its installation is as easy as:
1 | $ pip install tidehunter |
The repository is on GitHub with sufficient examples to get you started.