What is Observability

• Metrics: Prometheus, Grafana, Datadog, New Relic. Aggregated, efficient
• Logs: Loki, ELK stack, CloudWatch. Full-text, expensive at scale
• Traces: Jaeger, Zipkin, Tempo. Per-request detailed flow
• Correlation: trace_id links all 3 (standardised via OpenTelemetry)
• Cardinality explosion: high-cardinality labels (user_id) kill Prometheus

// OpenTelemetry instrumented code
const tracer = trace.getTracer('my-app');
const span = tracer.startSpan('db-query');
try {
  await db.query('SELECT ...')
} finally {
  span.end();  // exports trace to Jaeger/Tempo
}

• OpenTelemetry
• SLI/SLO

Metrics are quantitative measurements that provide insights into the performance and health of a system. In the realm of observability, metrics play a crucial role as they allow teams to track the state of various components over time. Metrics can be categorized into two types: counter and gauge.

A counter is a cumulative metric that only increases, such as the number of requests served by a server. In contrast, a gauge can go up and down, such as the current CPU usage.

Examples of key metrics include:

• CPU Utilization: Measures the percentage of CPU capacity being used.
• Requests Per Second (RPS): Indicates the number of requests being handled by the server per second.
• Error Rate: The percentage of requests that result in an error.

To effectively leverage metrics, organizations often utilize tools like Prometheus or Grafana for collection and visualization. For instance, to monitor CPU usage with Prometheus, you can use the following configuration in your prometheus.yml file:

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

This configuration allows Prometheus to scrape metrics from the Node Exporter, which collects system-level metrics such as CPU utilization. By analyzing these metrics, teams can quickly identify performance bottlenecks and make data-driven decisions.

Logs are an essential component of observability, providing detailed records of events that occur within a system. They are invaluable for troubleshooting and understanding the behavior of applications in real-time. Unlike metrics, which provide a high-level overview, logs offer granular detail that can help in diagnosing issues.

Logs typically contain timestamps, severity levels (e.g., info, warning, error), and contextual information about the event. This information can help teams trace the source of problems and understand the context in which they occurred.

Common log types include:

• Application Logs: Generated by the application to record its operations, including errors and transactions.
• Server Logs: Generated by servers to track access and usage patterns, often used for security audits.
• Audit Logs: Record changes made to the system, providing a trail for compliance and security investigations.

To implement effective logging, developers often use logging frameworks like Log4j for Java applications or winston for Node.js. For instance, a basic configuration in a Node.js application using winston might look like this:

const winston = require('winston');
const logger = winston.createLogger({
  level: 'info',
  format: winston.format.json(),
  transports: [
    new winston.transports.File({ filename: 'combined.log' }),
    new winston.transports.Console() 
  ]
});

This setup allows logging to both a file and the console, ensuring that all events are captured for later analysis. By leveraging logs effectively, organizations can gain deeper insights into their systems and respond to incidents more swiftly.

Traces are a vital aspect of observability, particularly in distributed systems where requests traverse multiple services. Tracing allows teams to visualize the path a request takes through various components, helping to identify latency issues and bottlenecks. By capturing traces, organizations can gain insights into how different services interact and how delays accumulate across the system.

A key concept in tracing is distributed tracing, which involves tracking requests as they propagate through multiple services. This is especially important in microservices architectures, where a single user request may involve several services communicating with one another.

Common tracing tools include:

• OpenTracing: A vendor-neutral API for distributed tracing.
• Jaeger: An open-source tool for monitoring and troubleshooting microservices.
• Zipkin: A distributed tracing system that helps gather timing data.

To implement distributed tracing, you can use OpenTracing with Jaeger. Below is an example of a basic setup in a Node.js application:

const { initTracer } = require('jaeger-client');
const options = {
  serviceName: 'my-service',
};
const tracer = initTracer(options, {});
const span = tracer.startSpan('my-operation');
span.finish();

This code initializes a Jaeger tracer for the service and creates a span representing an operation. By instrumenting your application with tracing, you can better understand request flows and pinpoint areas for optimization, ultimately enhancing the overall performance and reliability of your system.