Description: One of the most important part of Hadoop technology stack - distributed filesystem. Used as main storage for processed data and as source for different ML task. Used also as main storage for Hive warehouse

Provision script (Puppet manifest): hadoop.pp

Additional info: Hadoop site

---

Table of content:

• Home
• Prerequisites
• Developement stand provisioning
• Components
  • Crawler (Spring Boot, Java)
  • Message Broker (Kafka)
  • Distributed file storage (Hadoop’s HDFS)
  • Data warehouse (Apache Hive)
  • Distributed business logic cluster (Akka, Scala)
  • Distributed data processing cluster (Spark, Scala)
  • Indexing/Search engine (Elasticsearch)
  • REST server (Lagom, Scala)
  • Service coordination (Zookeeper)
  • Time-series database (monitoring data) (InfluxDB)
  • Metrics collector (Telegraf)
  • Monitoring visualization service (Grafana)
  • Reverse proxy/load-balancer (nginx)
• Monitoring Links
• Development
  • Build Tool
  • Provisioning (DevOps specific)
  • Configuration
  • Continuous Integration, Continuous Deployment
  • Implementation design
  • Logging
  • Metrics
  • Used ports
  • Adding new server