Tech Scouter

Introduction This is about my experience with Celery a distributed framework in Python in heavy workload environment.  I am dividing the blog into the different sections: Product Brief Current Architecture Problem With Data loads and Sleepless Nights and  Solutions Tried The Final Destination Product Brief We are working on a product that aims at fetching data from the multiple sources and aggregating that data to generate insights from that data. Some of the Data Sources that we support as of now are: Stock News from multiple sources RSS feed  Twitter Reddit Yahoo News Yahoo Finance Earning Calenders Report Filings Company Financials Stock Prices Current Architecture Broad View We knew that problem that we are solving has to deal with the cruel decentralized Internet. And we need to divide the large task of getting the data from the web and analyzing it into small tasks. Fig 1 On exploring different projects and technologies and analyzing the community support we cam

Introduction Firstly, a brief background about myself. I am working as a Software Engineer in one of the Alternate Asset Management Organization ( Handling 1.4 Trillion with our product suite ) responsible for maintaining and developing a product ALT Data Analyzer. My work is focused on making the engine run and feed the machines with their food. This article explains the problems we faced with scaling up our architecture and solution we followed.  I am dividing the blog in the following different sections: Product Brief Current Architecture Problem With Data loads and Sleepless Nights Solutions Tried The Final Destination Product Brief The idea of building this product was to give users an aggregated view of the WEB around a company. By WEB I mean the data that is flowing freely over all the decentralized nodes of the internet. We try to capture all the financial, technical and fundamental data for the companies, pass that data through our massaging and analyz