BlogUnderstanding and Adding Value to the Elastic Stack

data-storage-2-800x400

The Elastic Stack is designed and engineered following 3 core principles:  Speed, Scalability, and Relevance. The Elastic Stack is a powerful & flexible analytics platform that allows for adoption across a large number of use cases such as:

  • logging
  • metrics analytics
  • security analytics
  • application performance monitoring (APM)
  • business analytics
  • site search
  • app search
  • enterprise search

At the heart of the Elastic Stack is Elasticsearch, which serves as the horizontally scalable distributed datastore and search engine that allows for the storage, search, and analysis of data in near real-time at the petabyte scale and beyond. Elasticsearch accepts most well-formed JSON data and offers a robust REST-based API for monitoring, management, and analysis. A dynamic schema is created for an index at time of ingestion and does not have to be explicitly defined, providing a flexible data model for quick iteration. Fields from this dynamic schema are indexed at ingestion, allowing for queries and aggregations that are executed on text or metric data to return in milliseconds.

In addition to Elasticsearch, the Elastic Stack includes:

Logstash – a server-side data processing pipeline that ingests data from a multitude of sources at the same time, transforms it, and then passes it along as JSON files to the Elasticsearch database.

Kibana – the UI layer on top of Elasticsearch that provides an analytics window into your development and operations activities. It allows you to view the data and navigate the Elastic Stack with interactive visualizations, graphs, and more.

Beats – these are the agents that send data from hundreds or thousands of devices back to Logstash and/or Elasticsearch.

The open-source nature of the Elastic Stack can create challenges, especially with regard to development and time to value. To help ease these concerns, the team here at ASG have created some enhanced offerings and plug-ins, including:

  • Automation technology
  • Database connectors and interfaces to connect to more traditional data warehouses
  • Custom-coded Ansible modules to retrieve, manage, and compile standard device log files and device configurations

With these additions to the Elastic Stack, we can take you from data creation (log and config files) to data collection (via Ansible modules), to analysis (Elasticsearch and ML), to archive (Cloudian object store), and finally to the cloud itself (glacier storage).

You can utilize our iLab Services to check out how Elastic Stack and ASG together can make all the difference. Our iLab offers you an enterprise-grade infrastructure where you can evaluate technology solutions in a practical setting to see how they address your challenges. It’s a great way to make informed decisions.

About the Author

Dustin Smith

Dustin Smith, Chief Technologist

Throughout his twenty-five year career, Dustin Smith has specialized in designing enterprise architectural solutions. As the Chief Technologist at ASG, Dustin uses his advanced understanding of cloud compute models to help customers develop and align their cloud strategies with their business objectives. His master-level engineering knowledge spans storage, systems, and networking.