Self-documenting runtime: becoming omniscient with Contexture
Authors: Alexander Kouznetsov
Analysis by print statements and logs - we've all done it. Now our code is all grown up, crunching big data on distributed clusters. How can you know what each of your many processes is doing right now? What about the last 10 hours? I have developed a technique and a toolkit for granular realtime monitoring and analysis of runtime Python objects that requires minimal effort to both integrate these capabilities your code and scale. The flexible and database-/transport-agnostic nature of the toolkit lead to various possible use cases, such as dataflow systems - I will cover this and some others.
This talk explores structured logging, profiling, introspection and object streaming; how write code to do that automatically using Contexture and what one can do with the data afterwards. We will see examples using RabbitMQ and ElasticSearch.