If you want to compile and run this project on your own machine/server you need to install such software:

• Docker (containerization service) (installation instructions)
• Docker Composer (tool for defining and running multi-container Docker applications) (installation instructions)
• Java 8 (main platform (JVM) for code execution)

Code was mainly produced and tested only on Linux platform - so, please, use similar platform for your work with this project.

---

Table of content:

• Home
• Prerequisites
• Developement stand provisioning
• Components
  • Crawler (Spring Boot, Java)
  • Message Broker (Kafka)
  • Distributed file storage (Hadoop’s HDFS)
  • Data warehouse (Apache Hive)
  • Distributed business logic cluster (Akka, Scala)
  • Distributed data processing cluster (Spark, Scala)
  • Indexing/Search engine (Elasticsearch)
  • REST server (Lagom, Scala)
  • Service coordination (Zookeeper)
  • Time-series database (monitoring data) (InfluxDB)
  • Metrics collector (Telegraf)
  • Monitoring visualization service (Grafana)
  • Reverse proxy/load-balancer (nginx)
• Monitoring Links
• Development
  • Build Tool
  • Provisioning (DevOps specific)
  • Configuration
  • Continuous Integration, Continuous Deployment
  • Implementation design
  • Logging
  • Metrics
  • Used ports
  • Adding new server