Tengu is a platform for big data experimentation, which allows for scalable streaming, analysis and storage of large amounts of heterogeneous data. Tengu offers access to heterogeneous storage technologies, supports both offline and real-time data analysis components and provides resource and data monitoring tools. Experiment setup configuration is fully automated.
The Tengu architecture consists of a front end and the Tengu Core. The front end translates RESTful method calls into resource specifications for modular Tengu Core instantiations, so that experimenters can built their own Tengu platform according to their specific needs.
- support for Lambda architecture analysis (aggregated batch and speed computing)
- batch analysis via Apache Hadoop/Spark
- stream processing via Apache Storm
- application virtualization via private OpenStack clouds
- support for data stores such as MySQL, ElasticSearch and Cassandra
- live data store transformation
Extensive platform documentation is available online (http://tengu.intec.ugent.be). A tutorial is available on the FGRE 2015 website (http://doc.ilabt.iminds.be/fgre/). Tengu makes use of the jFed CLI tool (http://jfed.iminds.be/) and is instantiated on the iMinds Virtual Wall.
Tengu has been used by experimenters to analyse data in a wide variety of application domains:
- abuse detection on social media sites
- recommendations for and monitoring of large crowds in festivals
- security log analysis and management
- automated real-time classification of news items