sa-lessons/modules/module5/theory.md

Модуль 5: Мониторинг и наблюдаемость

Содержание модуля

1. Основы мониторинга
2. Prometheus и Grafana
3. Логирование
4. Distributed Tracing
5. Observability
6. Alerting
7. SLI/SLO/SLA

---

Основы мониторинга

Что такое мониторинг

Мониторинг — процесс сбора, анализа и использования информации о работе системы для обеспечения её стабильности и производительности.

Цели мониторинга

1. Проактивное обнаружение проблем

• Выявление проблем до их влияния на пользователей
• Предотвращение каскадных отказов
• Оптимизация производительности

2. Реактивное решение проблем

• Быстрая диагностика инцидентов
• Сокращение времени восстановления (MTTR)
• Анализ причин отказов

3. Бизнес-аналитика

• Понимание поведения пользователей
• Анализ производительности бизнес-процессов
• Планирование ресурсов

Типы мониторинга

1. Infrastructure Monitoring

• CPU, Memory, Disk, Network
• Контейнеры и виртуальные машины
• Базы данных и кэши

2. Application Monitoring

• Метрики приложения
• Производительность кода
• Ошибки и исключения

3. Business Monitoring

• Ключевые бизнес-метрики
• Пользовательская активность
• Конверсии и доходы

Метрики мониторинга

1. Counters (Счетчики)

Определение: Метрики, которые только увеличиваются.

Технический пример - Advanced Metrics Collection:

@Component
public class RequestCounter {
    private final Counter requestCounter;
    private final Counter errorCounter;
    private final Timer requestTimer;
    private final MeterRegistry meterRegistry;
    
    public RequestCounter(MeterRegistry meterRegistry) {
        this.meterRegistry = meterRegistry;
        
        this.requestCounter = Counter.builder("http.requests.total")
            .description("Total number of HTTP requests")
            .tag("service", "user-service")
            .register(meterRegistry);
            
        this.errorCounter = Counter.builder("http.errors.total")
            .description("Total number of HTTP errors")
            .tag("service", "user-service")
            .register(meterRegistry);
            
        this.requestTimer = Timer.builder("http.request.duration")
            .description("HTTP request duration")
            .tag("service", "user-service")
            .register(meterRegistry);
    }
    
    public void increment(String method, String endpoint, String status) {
        requestCounter.increment(
            Tags.of(
                "method", method,
                "endpoint", endpoint,
                "status", status
            )
        );
        
        if (Integer.parseInt(status) >= 400) {
            errorCounter.increment(
                Tags.of(
                    "method", method,
                    "endpoint", endpoint,
                    "error_type", getErrorType(status)
                )
            );
        }
    }
    
    public Timer.Sample startTimer() {
        return Timer.start(meterRegistry);
    }
    
    public void recordDuration(Timer.Sample sample, String method, String endpoint) {
        sample.stop(requestTimer.tags(
            "method", method,
            "endpoint", endpoint
        ));
    }
    
    private String getErrorType(String status) {
        int statusCode = Integer.parseInt(status);
        if (statusCode >= 400 && statusCode < 500) {
            return "client_error";
        } else if (statusCode >= 500) {
            return "server_error";
        }
        return "unknown";
    }
}

// Custom Metrics Service
@Service
public class MetricsService {
    
    private final MeterRegistry meterRegistry;
    private final Counter businessCounter;
    private final Gauge activeUsersGauge;
    private final DistributionSummary orderValueSummary;
    
    public MetricsService(MeterRegistry meterRegistry) {
        this.meterRegistry = meterRegistry;
        
        this.businessCounter = Counter.builder("business.events.total")
            .description("Business events counter")
            .register(meterRegistry);
            
        this.activeUsersGauge = Gauge.builder("active.users")
            .description("Number of active users")
            .register(meterRegistry, this, MetricsService::getActiveUsersCount);
            
        this.orderValueSummary = DistributionSummary.builder("order.value")
            .description("Order value distribution")
            .baseUnit("currency")
            .register(meterRegistry);
    }
    
    public void recordBusinessEvent(String eventType, String userId) {
        businessCounter.increment(
            Tags.of(
                "event_type", eventType,
                "user_id", userId
            )
        );
    }
    
    public void recordOrderValue(BigDecimal value, String currency) {
        orderValueSummary.record(value.doubleValue());
        
        // Record currency-specific metric
        Counter.builder("order.value.by.currency")
            .tag("currency", currency)
            .register(meterRegistry)
            .increment(value.doubleValue());
    }
    
    private double getActiveUsersCount() {
        // Implementation to get active users count
        return 0.0; // Placeholder
    }
}

// Metrics Configuration
@Configuration
@EnableConfigurationProperties(MetricsProperties.class)
public class MetricsConfig {
    
    @Bean
    public MeterRegistry meterRegistry() {
        return new SimpleMeterRegistry();
    }
    
    @Bean
    public PrometheusMeterRegistry prometheusMeterRegistry() {
        return new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
    }
    
    @Bean
    public CompositeMeterRegistry compositeMeterRegistry(
            MeterRegistry simpleRegistry,
            PrometheusMeterRegistry prometheusRegistry) {
        
        CompositeMeterRegistry composite = new CompositeMeterRegistry();
        composite.add(simpleRegistry);
        composite.add(prometheusRegistry);
        
        return composite;
    }
    
    @Bean
    public TimedAspect timedAspect(MeterRegistry meterRegistry) {
        return new TimedAspect(meterRegistry);
    }
    
    @Bean
    public CountedAspect countedAspect(MeterRegistry meterRegistry) {
        return new CountedAspect(meterRegistry);
    }
}

// AOP for automatic metrics
@Aspect
@Component
public class MetricsAspect {
    
    private final MeterRegistry meterRegistry;
    private final Timer.Sample sample;
    
    public MetricsAspect(MeterRegistry meterRegistry) {
        this.meterRegistry = meterRegistry;
    }
    
    @Around("@annotation(org.springframework.web.bind.annotation.GetMapping)")
    public Object timeGetRequests(ProceedingJoinPoint joinPoint) throws Throwable {
        Timer.Sample sample = Timer.start(meterRegistry);
        try {
            Object result = joinPoint.proceed();
            sample.stop(Timer.builder("http.get.requests")
                .tag("endpoint", getEndpointName(joinPoint))
                .register(meterRegistry));
            return result;
        } catch (Exception e) {
            sample.stop(Timer.builder("http.get.requests")
                .tag("endpoint", getEndpointName(joinPoint))
                .tag("error", "true")
                .register(meterRegistry));
            throw e;
        }
    }
    
    private String getEndpointName(ProceedingJoinPoint joinPoint) {
        Method method = ((MethodSignature) joinPoint.getSignature()).getMethod();
        return method.getDeclaringClass().getSimpleName() + "." + method.getName();
    }
}

Node.js примеры - метрики с prom-client:

import express, { Request, Response, NextFunction } from 'express';
import client from 'prom-client';
import { PrismaClient } from '@prisma/client';

const app = express();
const prisma = new PrismaClient();

// Регистр метрик
const register = new client.Registry();

// Добавление стандартных метрик Node.js
client.collectDefaultMetrics({ register });

// Кастомные метрики
const httpRequestDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.1, 0.3, 0.5, 0.7, 1, 3, 5, 7, 10]
});

const httpRequestTotal = new client.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['method', 'route', 'status_code']
});

const activeConnections = new client.Gauge({
  name: 'active_connections',
  help: 'Number of active connections'
});

const businessEvents = new client.Counter({
  name: 'business_events_total',
  help: 'Total number of business events',
  labelNames: ['event_type', 'status']
});

const orderValue = new client.Histogram({
  name: 'order_value_dollars',
  help: 'Order value in dollars',
  buckets: [10, 50, 100, 500, 1000, 5000, 10000]
});

// Регистрация метрик
register.registerMetric(httpRequestDuration);
register.registerMetric(httpRequestTotal);
register.registerMetric(activeConnections);
register.registerMetric(businessEvents);
register.registerMetric(orderValue);

// Middleware для автоматического сбора метрик
app.use((req: Request, res: Response, next: NextFunction) => {
  const start = Date.now();
  
  res.on('finish', () => {
    const duration = (Date.now() - start) / 1000;
    const route = req.route?.path || req.path;
    
    httpRequestDuration
      .labels(req.method, route, res.statusCode.toString())
      .observe(duration);
    
    httpRequestTotal
      .labels(req.method, route, res.statusCode.toString())
      .inc();
  });
  
  next();
});

// Service для работы с метриками
class MetricsService {
  private activeConnectionsCount = 0;

  incrementActiveConnections(): void {
    this.activeConnectionsCount++;
    activeConnections.set(this.activeConnectionsCount);
  }

  decrementActiveConnections(): void {
    this.activeConnectionsCount--;
    activeConnections.set(this.activeConnectionsCount);
  }

  recordBusinessEvent(eventType: string, status: string): void {
    businessEvents.labels(eventType, status).inc();
  }

  recordOrderValue(value: number): void {
    orderValue.observe(value);
  }
}

// Order Service с метриками
class OrderService {
  constructor(
    private orderRepository: OrderRepository,
    private metricsService: MetricsService
  ) {}

  async createOrder(request: CreateOrderRequest): Promise<Order> {
    const timer = httpRequestDuration.startTimer();
    
    try {
      // Бизнес-логика
      const order = await this.orderRepository.save({
        userId: request.userId,
        amount: request.amount,
        status: 'CREATED',
        createdAt: new Date()
      });

      // Запись метрик
      this.metricsService.recordBusinessEvent('order_created', 'success');
      this.metricsService.recordOrderValue(request.amount);

      return order;
    } catch (error) {
      this.metricsService.recordBusinessEvent('order_created', 'error');
      throw error;
    } finally {
      timer({ method: 'POST', route: '/api/orders', status_code: '200' });
    }
  }

  async getOrder(id: number): Promise<Order | null> {
    const timer = httpRequestDuration.startTimer();
    
    try {
      const order = await this.orderRepository.findById(id);
      return order;
    } finally {
      timer({ method: 'GET', route: '/api/orders/:id', status_code: '200' });
    }
  }
}

// Repository
interface OrderRepository {
  save(order: Omit<Order, 'id'>): Promise<Order>;
  findById(id: number): Promise<Order | null>;
}

class PrismaOrderRepository implements OrderRepository {
  async save(order: Omit<Order, 'id'>): Promise<Order> {
    return await prisma.order.create({ data: order });
  }

  async findById(id: number): Promise<Order | null> {
    return await prisma.order.findUnique({ where: { id } });
  }
}

// Dependency injection
const metricsService = new MetricsService();
const orderRepository = new PrismaOrderRepository();
const orderService = new OrderService(orderRepository, metricsService);

// API endpoints
app.post('/api/orders', async (req: Request, res: Response, next: NextFunction) => {
  try {
    const order = await orderService.createOrder(req.body);
    res.json(order);
  } catch (error) {
    next(error);
  }
});

app.get('/api/orders/:id', async (req: Request, res: Response, next: NextFunction) => {
  try {
    const id = parseInt(req.params.id);
    const order = await orderService.getOrder(id);
    
    if (!order) {
      res.status(404).json({ error: 'Order not found' });
      return;
    }
    
    res.json(order);
  } catch (error) {
    next(error);
  }
});

// Endpoint для метрик Prometheus
app.get('/metrics', async (req: Request, res: Response) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

// Health check endpoint
app.get('/health', (req: Request, res: Response) => {
  res.json({ status: 'healthy', timestamp: new Date().toISOString() });
});

// Middleware для отслеживания активных соединений
app.use((req: Request, res: Response, next: NextFunction) => {
  metricsService.incrementActiveConnections();
  
  res.on('finish', () => {
    metricsService.decrementActiveConnections();
  });
  
  next();
});

// Error handling middleware
app.use((error: Error, req: Request, res: Response, next: NextFunction) => {
  console.error(error);
  res.status(500).json({ error: 'Internal server error' });
});

app.listen(3000, () => {
  console.log('Server running on port 3000');
  console.log('Metrics available at http://localhost:3000/metrics');
});

Node.js примеры с библиотекой prom-client для NestJS:

import { Module } from '@nestjs/common';
import { PrometheusModule } from '@willsoto/nestjs-prometheus';
import { Controller, Get, Post, Body, Param } from '@nestjs/common';
import { Counter, Histogram, Gauge, register } from 'prom-client';

@Module({
  imports: [
    PrometheusModule.register({
      defaultMetrics: {
        enabled: true,
      },
    }),
  ],
})
export class AppModule {}

// Кастомные метрики
const httpRequestDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.1, 0.3, 0.5, 0.7, 1, 3, 5, 7, 10],
});

const businessEvents = new Counter({
  name: 'business_events_total',
  help: 'Total number of business events',
  labelNames: ['event_type', 'status'],
});

const activeUsers = new Gauge({
  name: 'active_users',
  help: 'Number of active users',
});

// Регистрация метрик
register.registerMetric(httpRequestDuration);
register.registerMetric(businessEvents);
register.registerMetric(activeUsers);

@Controller('orders')
export class OrderController {
  constructor(private orderService: OrderService) {}

  @Post()
  async createOrder(@Body() createOrderDto: CreateOrderDto) {
    const timer = httpRequestDuration.startTimer();
    
    try {
      const order = await this.orderService.createOrder(createOrderDto);
      
      // Запись бизнес-метрик
      businessEvents.labels('order_created', 'success').inc();
      
      return order;
    } catch (error) {
      businessEvents.labels('order_created', 'error').inc();
      throw error;
    } finally {
      timer({ method: 'POST', route: '/orders', status_code: '200' });
    }
  }

  @Get(':id')
  async getOrder(@Param('id') id: string) {
    const timer = httpRequestDuration.startTimer();
    
    try {
      const order = await this.orderService.getOrder(+id);
      return order;
    } finally {
      timer({ method: 'GET', route: '/orders/:id', status_code: '200' });
    }
  }
}

@Controller('metrics')
export class MetricsController {
  @Get()
  async getMetrics() {
    return register.metrics();
  }
}

Rust примеры - Метрики с prometheus:

// Метрики с prometheus для Actix Web
use actix_web::{web, App, HttpServer, middleware::Logger};
use prometheus::{Counter, Histogram, Gauge, Registry, Encoder, TextEncoder};
use std::sync::Arc;
use std::time::Instant;

// Глобальные метрики
lazy_static::lazy_static! {
    static ref HTTP_REQUESTS_TOTAL: Counter = Counter::new(
        "http_requests_total",
        "Total number of HTTP requests"
    ).unwrap();
    
    static ref HTTP_REQUEST_DURATION: Histogram = Histogram::new(
        "http_request_duration_seconds",
        "Duration of HTTP requests in seconds"
    ).unwrap();
    
    static ref ACTIVE_CONNECTIONS: Gauge = Gauge::new(
        "active_connections",
        "Number of active connections"
    ).unwrap();
    
    static ref BUSINESS_EVENTS_TOTAL: Counter = Counter::new(
        "business_events_total",
        "Total number of business events"
    ).unwrap();
    
    static ref ORDER_VALUE: Histogram = Histogram::new(
        "order_value_dollars",
        "Order value in dollars"
    ).unwrap();
}

// Метрики сервис
pub struct MetricsService {
    registry: Registry,
    active_connections_count: Arc<std::sync::atomic::AtomicUsize>,
}

impl MetricsService {
    pub fn new() -> Self {
        let registry = Registry::new();
        
        // Регистрация метрик
        registry.register(Box::new(HTTP_REQUESTS_TOTAL.clone())).unwrap();
        registry.register(Box::new(HTTP_REQUEST_DURATION.clone())).unwrap();
        registry.register(Box::new(ACTIVE_CONNECTIONS.clone())).unwrap();
        registry.register(Box::new(BUSINESS_EVENTS_TOTAL.clone())).unwrap();
        registry.register(Box::new(ORDER_VALUE.clone())).unwrap();
        
        Self {
            registry,
            active_connections_count: Arc::new(std::sync::atomic::AtomicUsize::new(0)),
        }
    }
    
    pub fn increment_active_connections(&self) {
        let count = self.active_connections_count.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
        ACTIVE_CONNECTIONS.set(count as f64 + 1.0);
    }
    
    pub fn decrement_active_connections(&self) {
        let count = self.active_connections_count.fetch_sub(1, std::sync::atomic::Ordering::SeqCst);
        ACTIVE_CONNECTIONS.set(count as f64 - 1.0);
    }
    
    pub fn record_business_event(&self, event_type: &str, status: &str) {
        BUSINESS_EVENTS_TOTAL.with_label_values(&[event_type, status]).inc();
    }
    
    pub fn record_order_value(&self, value: f64) {
        ORDER_VALUE.observe(value);
    }
    
    pub fn get_metrics(&self) -> String {
        let mut buffer = Vec::new();
        let encoder = TextEncoder::new();
        let metric_families = self.registry.gather();
        encoder.encode(&metric_families, &mut buffer).unwrap();
        String::from_utf8(buffer).unwrap()
    }
}

// Order Service с метриками
pub struct OrderService {
    metrics_service: Arc<MetricsService>,
}

impl OrderService {
    pub fn new(metrics_service: Arc<MetricsService>) -> Self {
        Self { metrics_service }
    }
    
    pub async fn create_order(&self, request: CreateOrderRequest) -> Result<Order, String> {
        let timer = HTTP_REQUEST_DURATION.start_timer();
        
        // Бизнес-логика
        let order = Order {
            id: uuid::Uuid::new_v4(),
            user_id: request.user_id,
            amount: request.amount,
            status: "CREATED".to_string(),
            created_at: chrono::Utc::now(),
        };
        
        // Запись метрик
        self.metrics_service.record_business_event("order_created", "success");
        self.metrics_service.record_order_value(request.amount);
        
        timer.observe_duration(std::time::Duration::from_millis(100));
        
        Ok(order)
    }
    
    pub async fn get_order(&self, id: uuid::Uuid) -> Result<Option<Order>, String> {
        let timer = HTTP_REQUEST_DURATION.start_timer();
        
        // Бизнес-логика
        let order = Some(Order {
            id,
            user_id: 1,
            amount: 100.0,
            status: "CREATED".to_string(),
            created_at: chrono::Utc::now(),
        });
        
        timer.observe_duration(std::time::Duration::from_millis(50));
        
        Ok(order)
    }
}

// Структуры данных
#[derive(serde::Deserialize)]
pub struct CreateOrderRequest {
    pub user_id: u64,
    pub amount: f64,
}

#[derive(serde::Serialize)]
pub struct Order {
    pub id: uuid::Uuid,
    pub user_id: u64,
    pub amount: f64,
    pub status: String,
    pub created_at: chrono::DateTime<chrono::Utc>,
}

// Handlers
async fn create_order(
    order_service: web::Data<Arc<OrderService>>,
    request: web::Json<CreateOrderRequest>,
) -> actix_web::Result<actix_web::HttpResponse> {
    HTTP_REQUESTS_TOTAL.with_label_values(&["POST", "/api/orders", "200"]).inc();
    
    match order_service.create_order(request.into_inner()).await {
        Ok(order) => Ok(actix_web::HttpResponse::Ok().json(order)),
        Err(e) => Ok(actix_web::HttpResponse::BadRequest().json(serde_json::json!({
            "error": e
        }))),
    }
}

async fn get_order(
    order_service: web::Data<Arc<OrderService>>,
    path: web::Path<uuid::Uuid>,
) -> actix_web::Result<actix_web::HttpResponse> {
    let id = path.into_inner();
    HTTP_REQUESTS_TOTAL.with_label_values(&["GET", "/api/orders/:id", "200"]).inc();
    
    match order_service.get_order(id).await {
        Ok(Some(order)) => Ok(actix_web::HttpResponse::Ok().json(order)),
        Ok(None) => Ok(actix_web::HttpResponse::NotFound().json(serde_json::json!({
            "error": "Order not found"
        }))),
        Err(e) => Ok(actix_web::HttpResponse::BadRequest().json(serde_json::json!({
            "error": e
        }))),
    }
}

async fn metrics(metrics_service: web::Data<Arc<MetricsService>>) -> actix_web::Result<actix_web::HttpResponse> {
    let metrics_data = metrics_service.get_metrics();
    Ok(actix_web::HttpResponse::Ok()
        .content_type("text/plain")
        .body(metrics_data))
}

async fn health() -> actix_web::Result<actix_web::HttpResponse> {
    Ok(actix_web::HttpResponse::Ok().json(serde_json::json!({
        "status": "healthy",
        "timestamp": chrono::Utc::now().to_rfc3339()
    })))
}

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    let metrics_service = Arc::new(MetricsService::new());
    let order_service = Arc::new(OrderService::new(metrics_service.clone()));
    
    HttpServer::new(move || {
        App::new()
            .app_data(web::Data::new(order_service.clone()))
            .app_data(web::Data::new(metrics_service.clone()))
            .wrap(Logger::default())
            .route("/api/orders", web::post().to(create_order))
            .route("/api/orders/{id}", web::get().to(get_order))
            .route("/metrics", web::get().to(metrics))
            .route("/health", web::get().to(health))
    })
    .bind("127.0.0.1:3000")?
    .run()
    .await
}

Python примеры - Метрики с prometheus:

# Метрики с prometheus для FastAPI
from fastapi import FastAPI, Request, Response
from prometheus_client import Counter, Histogram, Gauge, generate_latest, CONTENT_TYPE_LATEST
import time
import asyncio
from typing import Dict, Any
import uuid
from datetime import datetime

app = FastAPI(title="Metrics Service", version="1.0.0")

# Метрики
HTTP_REQUESTS_TOTAL = Counter(
    'http_requests_total',
    'Total number of HTTP requests',
    ['method', 'route', 'status_code']
)

HTTP_REQUEST_DURATION = Histogram(
    'http_request_duration_seconds',
    'Duration of HTTP requests in seconds',
    ['method', 'route', 'status_code'],
    buckets=[0.1, 0.3, 0.5, 0.7, 1, 3, 5, 7, 10]
)

ACTIVE_CONNECTIONS = Gauge(
    'active_connections',
    'Number of active connections'
)

BUSINESS_EVENTS_TOTAL = Counter(
    'business_events_total',
    'Total number of business events',
    ['event_type', 'status']
)

ORDER_VALUE = Histogram(
    'order_value_dollars',
    'Order value in dollars',
    buckets=[10, 50, 100, 500, 1000, 5000, 10000]
)

# Метрики сервис
class MetricsService:
    def __init__(self):
        self.active_connections_count = 0
    
    def increment_active_connections(self):
        self.active_connections_count += 1
        ACTIVE_CONNECTIONS.set(self.active_connections_count)
    
    def decrement_active_connections(self):
        self.active_connections_count -= 1
        ACTIVE_CONNECTIONS.set(self.active_connections_count)
    
    def record_business_event(self, event_type: str, status: str):
        BUSINESS_EVENTS_TOTAL.labels(event_type=event_type, status=status).inc()
    
    def record_order_value(self, value: float):
        ORDER_VALUE.observe(value)

# Order Service с метриками
class OrderService:
    def __init__(self, metrics_service: MetricsService):
        self.metrics_service = metrics_service
    
    async def create_order(self, request: Dict[str, Any]) -> Dict[str, Any]:
        # Бизнес-логика
        order = {
            "id": str(uuid.uuid4()),
            "user_id": request["user_id"],
            "amount": request["amount"],
            "status": "CREATED",
            "created_at": datetime.utcnow().isoformat()
        }
        
        # Запись метрик
        self.metrics_service.record_business_event("order_created", "success")
        self.metrics_service.record_order_value(request["amount"])
        
        return order
    
    async def get_order(self, order_id: str) -> Dict[str, Any]:
        # Бизнес-логика
        order = {
            "id": order_id,
            "user_id": 1,
            "amount": 100.0,
            "status": "CREATED",
            "created_at": datetime.utcnow().isoformat()
        }
        
        return order

# Dependency injection
metrics_service = MetricsService()
order_service = OrderService(metrics_service)

# Middleware для автоматического сбора метрик
@app.middleware("http")
async def metrics_middleware(request: Request, call_next):
    start_time = time.time()
    
    response = await call_next(request)
    
    duration = time.time() - start_time
    route = request.url.path
    method = request.method
    status_code = str(response.status_code)
    
    HTTP_REQUEST_DURATION.labels(
        method=method,
        route=route,
        status_code=status_code
    ).observe(duration)
    
    HTTP_REQUESTS_TOTAL.labels(
        method=method,
        route=route,
        status_code=status_code
    ).inc()
    
    return response

# API endpoints
@app.post("/api/orders")
async def create_order(request: Dict[str, Any]):
    order = await order_service.create_order(request)
    return order

@app.get("/api/orders/{order_id}")
async def get_order(order_id: str):
    order = await order_service.get_order(order_id)
    return order

@app.get("/metrics")
async def metrics():
    return Response(
        generate_latest(),
        media_type=CONTENT_TYPE_LATEST
    )

@app.get("/health")
async def health():
    return {
        "status": "healthy",
        "timestamp": datetime.utcnow().isoformat()
    }

# Middleware для отслеживания активных соединений
@app.middleware("http")
async def active_connections_middleware(request: Request, call_next):
    metrics_service.increment_active_connections()
    
    try:
        response = await call_next(request)
        return response
    finally:
        metrics_service.decrement_active_connections()

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=3000)

2. Gauges (Измерители)

Определение: Метрики, которые могут увеличиваться и уменьшаться.

@Component
public class ActiveConnectionsGauge {
    private final AtomicInteger activeConnections = new AtomicInteger(0);
    private final Gauge activeConnectionsGauge;
    
    public ActiveConnectionsGauge(MeterRegistry meterRegistry) {
        this.activeConnectionsGauge = Gauge.builder("active.connections")
            .description("Number of active connections")
            .register(meterRegistry, this, obj -> obj.activeConnections.get());
    }
    
    public void incrementConnections() {
        activeConnections.incrementAndGet();
    }
    
    public void decrementConnections() {
        activeConnections.decrementAndGet();
    }
}

3. Histograms (Гистограммы)

Определение: Метрики для измерения распределения значений.

@Component
public class ResponseTimeHistogram {
    private final Timer responseTimeTimer;
    
    public ResponseTimeHistogram(MeterRegistry meterRegistry) {
        this.responseTimeTimer = Timer.builder("http.request.duration")
            .description("HTTP request duration")
            .register(meterRegistry);
    }
    
    public void recordResponseTime(String endpoint, Duration duration) {
        responseTimeTimer.record(duration);
    }
}

4. Summaries (Сумматоры)

Определение: Метрики для измерения квантилей.

@Component
public class ResponseTimeSummary {
    private final DistributionSummary responseTimeSummary;
    
    public ResponseTimeSummary(MeterRegistry meterRegistry) {
        this.responseTimeSummary = DistributionSummary.builder("http.request.size")
            .description("HTTP request size in bytes")
            .register(meterRegistry);
    }
    
    public void recordRequestSize(long size) {
        responseTimeSummary.record(size);
    }
}

---

Prometheus и Grafana

Prometheus

Prometheus — система мониторинга и база данных временных рядов.

Архитектура Prometheus

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Application   │    │   Application   │    │   Application   │
│   (Metrics)     │    │   (Metrics)     │    │   (Metrics)     │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         └───────────────────────┼───────────────────────┘
                                 │
                    ┌─────────────────┐
                    │   Prometheus    │
                    │     Server      │
                    └─────────────────┘
                                 │
                    ┌─────────────────┐
                    │    Grafana      │
                    │   (Dashboard)   │
                    └─────────────────┘

PromQL (Язык запросов Prometheus)

Основные запросы:

# Простая метрика
http_requests_total

# Метрика с фильтрами
http_requests_total{method="GET", status="200"}

# Агрегация
sum(http_requests_total) by (method)

# Rate (скорость изменения)
rate(http_requests_total[5m])

# Histogram quantiles
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

Примеры запросов:

# Количество запросов в секунду
rate(http_requests_total[5m])

# Среднее время ответа
rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m])

# 95-й перцентиль времени ответа
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Ошибки в процентах
rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) * 100

Spring Boot + Prometheus

@Configuration
public class PrometheusConfig {
    @Bean
    public MeterRegistry meterRegistry() {
        return new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
    }
}

@RestController
public class MetricsController {
    private final MeterRegistry meterRegistry;
    
    public MetricsController(MeterRegistry meterRegistry) {
        this.meterRegistry = meterRegistry;
    }
    
    @GetMapping("/metrics")
    public String metrics() {
        return ((PrometheusMeterRegistry) meterRegistry).scrape();
    }
}

Grafana

Grafana — платформа для визуализации метрик и создания дашбордов.

Создание дашборда

{
  "dashboard": {
    "title": "Application Metrics",
    "panels": [
      {
        "title": "Request Rate",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(http_requests_total[5m])",
            "legendFormat": "{{method}} {{endpoint}}"
          }
        ]
      },
      {
        "title": "Response Time",
        "type": "graph",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))",
            "legendFormat": "95th percentile"
          }
        ]
      }
    ]
  }
}

Алерты в Grafana

{
  "alert": {
    "name": "High Error Rate",
    "message": "Error rate is above 5%",
    "conditions": [
      {
        "query": {
          "params": ["A", "5m", "now"]
        },
        "reducer": {
          "type": "last",
          "params": []
        },
        "evaluator": {
          "params": [0.05],
          "type": "gt"
        }
      }
    ]
  }
}

---

Логирование

Принципы логирования

1. Structured Logging

Определение: Логирование в структурированном формате (JSON).

Технический пример - Advanced Structured Logging:

@Slf4j
@RestController
public class OrderController {
    
    @Autowired
    private OrderService orderService;
    
    @Autowired
    private MetricsService metricsService;
    
    @PostMapping("/orders")
    public ResponseEntity<Order> createOrder(@RequestBody CreateOrderRequest request) {
        String correlationId = MDC.get("correlationId");
        String userId = MDC.get("userId");
        
        // Structured logging with business context
        log.info("Order creation started", 
            StructuredLogging.builder()
                .correlationId(correlationId)
                .userId(userId)
                .customerId(request.getCustomerId())
                .orderValue(request.getTotalValue())
                .itemsCount(request.getItems().size())
                .currency(request.getCurrency())
                .paymentMethod(request.getPaymentMethod())
                .build()
        );
        
        Timer.Sample sample = Timer.start();
        
        try {
            Order order = orderService.createOrder(request);
            
            // Success logging with performance metrics
            Duration duration = Duration.ofNanos(sample.stop());
            log.info("Order created successfully", 
                StructuredLogging.builder()
                    .correlationId(correlationId)
                    .userId(userId)
                    .orderId(order.getId())
                    .customerId(request.getCustomerId())
                    .status(order.getStatus())
                    .processingTimeMs(duration.toMillis())
                    .build()
            );
            
            // Record business metrics
            metricsService.recordBusinessEvent("order_created", userId);
            metricsService.recordOrderValue(request.getTotalValue(), request.getCurrency());
            
            return ResponseEntity.ok(order);
            
        } catch (ValidationException e) {
            log.warn("Order validation failed", 
                StructuredLogging.builder()
                    .correlationId(correlationId)
                    .userId(userId)
                    .customerId(request.getCustomerId())
                    .errorType("validation_error")
                    .errorMessage(e.getMessage())
                    .validationErrors(e.getValidationErrors())
                    .build()
            );
            return ResponseEntity.badRequest().body(new ErrorResponse(e.getMessage()));
            
        } catch (PaymentException e) {
            log.error("Payment processing failed", 
                StructuredLogging.builder()
                    .correlationId(correlationId)
                    .userId(userId)
                    .customerId(request.getCustomerId())
                    .errorType("payment_error")
                    .errorMessage(e.getMessage())
                    .paymentMethod(request.getPaymentMethod())
                    .orderValue(request.getTotalValue())
                    .build()
            );
            return ResponseEntity.status(HttpStatus.PAYMENT_REQUIRED)
                .body(new ErrorResponse("Payment processing failed"));
                
        } catch (Exception e) {
            log.error("Unexpected error during order creation", 
                StructuredLogging.builder()
                    .correlationId(correlationId)
                    .userId(userId)
                    .customerId(request.getCustomerId())
                    .errorType("system_error")
                    .errorMessage(e.getMessage())
                    .stackTrace(getStackTrace(e))
                    .build()
            );
            return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR)
                .body(new ErrorResponse("Internal server error"));
        }
    }
}

// Structured Logging Builder
public class StructuredLogging {
    private final Map<String, Object> fields = new HashMap<>();
    
    public static Builder builder() {
        return new Builder();
    }
    
    public static class Builder {
        private final StructuredLogging logging = new StructuredLogging();
        
        public Builder correlationId(String correlationId) {
            logging.fields.put("correlationId", correlationId);
            return this;
        }
        
        public Builder userId(String userId) {
            logging.fields.put("userId", userId);
            return this;
        }
        
        public Builder customerId(Long customerId) {
            logging.fields.put("customerId", customerId);
            return this;
        }
        
        public Builder orderId(Long orderId) {
            logging.fields.put("orderId", orderId);
            return this;
        }
        
        public Builder orderValue(BigDecimal value) {
            logging.fields.put("orderValue", value);
            return this;
        }
        
        public Builder itemsCount(Integer count) {
            logging.fields.put("itemsCount", count);
            return this;
        }
        
        public Builder currency(String currency) {
            logging.fields.put("currency", currency);
            return this;
        }
        
        public Builder paymentMethod(String method) {
            logging.fields.put("paymentMethod", method);
            return this;
        }
        
        public Builder status(String status) {
            logging.fields.put("status", status);
            return this;
        }
        
        public Builder processingTimeMs(Long timeMs) {
            logging.fields.put("processingTimeMs", timeMs);
            return this;
        }
        
        public Builder errorType(String errorType) {
            logging.fields.put("errorType", errorType);
            return this;
        }
        
        public Builder errorMessage(String message) {
            logging.fields.put("errorMessage", message);
            return this;
        }
        
        public Builder validationErrors(List<String> errors) {
            logging.fields.put("validationErrors", errors);
            return this;
        }
        
        public Builder stackTrace(String stackTrace) {
            logging.fields.put("stackTrace", stackTrace);
            return this;
        }
        
        public StructuredLogging build() {
            return logging;
        }
    }
    
    public Map<String, Object> getFields() {
        return fields;
    }
    
    @Override
    public String toString() {
        try {
            return new ObjectMapper().writeValueAsString(fields);
        } catch (Exception e) {
            return fields.toString();
        }
    }
}

// Advanced Correlation ID Filter
@Component
@Order(Ordered.HIGHEST_PRECEDENCE)
public class CorrelationIdFilter implements Filter {
    
    private static final String CORRELATION_ID_HEADER = "X-Correlation-ID";
    private static final String USER_ID_HEADER = "X-User-ID";
    private static final String REQUEST_ID_HEADER = "X-Request-ID";
    
    @Override
    public void doFilter(ServletRequest request, ServletResponse response, 
                        FilterChain chain) throws IOException, ServletException {
        
        HttpServletRequest httpRequest = (HttpServletRequest) request;
        HttpServletResponse httpResponse = (HttpServletResponse) response;
        
        // Extract or generate correlation ID
        String correlationId = httpRequest.getHeader(CORRELATION_ID_HEADER);
        if (correlationId == null || correlationId.isEmpty()) {
            correlationId = UUID.randomUUID().toString();
        }
        
        // Extract user ID
        String userId = httpRequest.getHeader(USER_ID_HEADER);
        
        // Generate request ID
        String requestId = UUID.randomUUID().toString();
        
        // Set MDC
        MDC.put("correlationId", correlationId);
        MDC.put("userId", userId);
        MDC.put("requestId", requestId);
        MDC.put("service", "order-service");
        MDC.put("version", "1.0.0");
        
        // Add headers to response
        httpResponse.setHeader(CORRELATION_ID_HEADER, correlationId);
        httpResponse.setHeader(REQUEST_ID_HEADER, requestId);
        
        try {
            chain.doFilter(request, response);
        } finally {
            MDC.clear();
        }
    }
}

// Logback Configuration for Structured Logging
// logback-spring.xml
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
    
    <!-- Console Appender with JSON format -->
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
            <providers>
                <timestamp/>
                <logLevel/>
                <loggerName/>
                <message/>
                <mdc/>
                <stackTrace/>
            </providers>
        </encoder>
    </appender>
    
    <!-- File Appender with JSON format -->
    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>/app/logs/order-service.json</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>/app/logs/order-service.%d{yyyy-MM-dd}.%i.json</fileNamePattern>
            <maxFileSize>100MB</maxFileSize>
            <maxHistory>30</maxHistory>
            <totalSizeCap>3GB</totalSizeCap>
        </rollingPolicy>
        <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
            <providers>
                <timestamp/>
                <logLevel/>
                <loggerName/>
                <message/>
                <mdc/>
                <stackTrace/>
            </providers>
        </encoder>
    </appender>
    
    <!-- Error-specific appender -->
    <appender name="ERROR_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>/app/logs/order-service-errors.json</file>
        <filter class="ch.qos.logback.classic.filter.LevelFilter">
            <level>ERROR</level>
            <onMatch>ACCEPT</onMatch>
            <onMismatch>DENY</onMismatch>
        </filter>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>/app/logs/order-service-errors.%d{yyyy-MM-dd}.%i.json</fileNamePattern>
            <maxFileSize>50MB</maxFileSize>
            <maxHistory>90</maxHistory>
        </rollingPolicy>
        <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
            <providers>
                <timestamp/>
                <logLevel/>
                <loggerName/>
                <message/>
                <mdc/>
                <stackTrace/>
            </providers>
        </encoder>
    </appender>
    
    <root level="INFO">
        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="FILE"/>
        <appender-ref ref="ERROR_FILE"/>
    </root>
</configuration>

2. Log Levels

• ERROR — критические ошибки
• WARN — предупреждения
• INFO — информационные сообщения
• DEBUG — отладочная информация
• TRACE — детальная трассировка

3. Correlation ID

@Component
public class CorrelationIdFilter implements Filter {
    @Override
    public void doFilter(ServletRequest request, ServletResponse response, 
                        FilterChain chain) throws IOException, ServletException {
        
        HttpServletRequest httpRequest = (HttpServletRequest) request;
        String correlationId = httpRequest.getHeader("X-Correlation-ID");
        
        if (correlationId == null) {
            correlationId = UUID.randomUUID().toString();
        }
        
        MDC.put("correlationId", correlationId);
        
        try {
            chain.doFilter(request, response);
        } finally {
            MDC.clear();
        }
    }
}

ELK Stack (Elasticsearch, Logstash, Kibana)

Конфигурация Logstash

# logstash.conf
input {
  beats {
    port => 5044
  }
}

filter {
  if [fields][service] == "order-service" {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}" }
    }
    
    date {
      match => [ "timestamp", "ISO8601" ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "logs-%{+YYYY.MM.dd}"
  }
}

Шаблон индекса Elasticsearch

{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1
    },
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "level": {
          "type": "keyword"
        },
        "message": {
          "type": "text"
        },
        "correlationId": {
          "type": "keyword"
        },
        "service": {
          "type": "keyword"
        }
      }
    }
  }
}

Fluentd

# fluent.conf
<source>
  @type tail
  path /var/log/application.log
  pos_file /var/log/fluentd.log.pos
  tag application.logs
  format json
</source>

<filter application.logs>
  @type record_transformer
  <record>
    service "order-service"
    environment "production"
  </record>
</filter>

<match application.logs>
  @type elasticsearch
  host elasticsearch
  port 9200
  index_name logs
  type_name _doc
</match>

---

Distributed Tracing

Что такое Distributed Tracing

Distributed Tracing — метод отслеживания запросов через распределенную систему.

OpenTelemetry

@Configuration
public class TracingConfig {
    @Bean
    public OpenTelemetry openTelemetry() {
        return OpenTelemetrySdk.builder()
            .setTracerProvider(
                SdkTracerProvider.builder()
                    .addSpanProcessor(BatchSpanProcessor.builder(
                        JaegerGrpcSpanExporter.builder()
                            .setEndpoint("http://jaeger:14250")
                            .build()
                    ).build())
                    .build()
            )
            .build();
    }
}

@Service
public class OrderService {
    private final Tracer tracer;
    
    public OrderService(OpenTelemetry openTelemetry) {
        this.tracer = openTelemetry.getTracer("order-service");
    }
    
    public Order createOrder(CreateOrderRequest request) {
        Span span = tracer.spanBuilder("create-order")
            .setAttribute("customer.id", request.getCustomerId())
            .setAttribute("order.value", request.getTotalValue().doubleValue())
            .startSpan();
        
        try (Scope scope = span.makeCurrent()) {
            // бизнес-логика
            Order order = processOrder(request);
            
            span.setStatus(StatusCode.OK);
            span.setAttribute("order.id", order.getId());
            
            return order;
        } catch (Exception e) {
            span.setStatus(StatusCode.ERROR, e.getMessage());
            span.recordException(e);
            throw e;
        } finally {
            span.end();
        }
    }
}

Jaeger

Конфигурация Jaeger

# docker-compose.yml
version: '3.8'
services:
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"
      - "14268:14268"
    environment:
      - COLLECTOR_OTLP_ENABLED=true

Анализ трассировок

@Component
public class TraceAnalyzer {
    public void analyzeTrace(String traceId) {
        // анализ трассировки
        List<Span> spans = getSpansByTraceId(traceId);
        
        // поиск узких мест
        Span slowestSpan = spans.stream()
            .max(Comparator.comparing(Span::getDuration))
            .orElse(null);
        
        // анализ ошибок
        List<Span> errorSpans = spans.stream()
            .filter(span -> span.getStatus().getStatusCode() == StatusCode.ERROR)
            .collect(Collectors.toList());
    }
}

---

Observability

Три столпа Observability

1. Metrics (Метрики)

• Количественные измерения
• Агрегированные данные
• Хорошо для мониторинга трендов

2. Logs (Логи)

• События с временными метками
• Детальная информация
• Хорошо для отладки

3. Traces (Трассировки)

• Путь запроса через систему
• Временные зависимости
• Хорошо для анализа производительности

Паттерны наблюдаемости

1. Health Checks

@Component
public class HealthIndicator implements HealthIndicator {
    @Override
    public Health health() {
        try {
            // проверка зависимостей
            boolean dbHealthy = checkDatabase();
            boolean cacheHealthy = checkCache();
            
            if (dbHealthy && cacheHealthy) {
                return Health.up()
                    .withDetail("database", "UP")
                    .withDetail("cache", "UP")
                    .build();
            } else {
                return Health.down()
                    .withDetail("database", dbHealthy ? "UP" : "DOWN")
                    .withDetail("cache", cacheHealthy ? "UP" : "DOWN")
                    .build();
            }
        } catch (Exception e) {
            return Health.down(e).build();
        }
    }
}

2. Custom Metrics

@Component
public class BusinessMetrics {
    private final Counter ordersCreated;
    private final Timer orderProcessingTime;
    private final Gauge activeUsers;
    
    public BusinessMetrics(MeterRegistry meterRegistry) {
        this.ordersCreated = Counter.builder("orders.created")
            .description("Total orders created")
            .register(meterRegistry);
            
        this.orderProcessingTime = Timer.builder("orders.processing.time")
            .description("Order processing time")
            .register(meterRegistry);
            
        this.activeUsers = Gauge.builder("users.active")
            .description("Number of active users")
            .register(meterRegistry, this, obj -> obj.getActiveUsersCount());
    }
    
    public void recordOrderCreated() {
        ordersCreated.increment();
    }
    
    public void recordOrderProcessingTime(Duration duration) {
        orderProcessingTime.record(duration);
    }
    
    private int getActiveUsersCount() {
        // логика подсчета активных пользователей
        return 0;
    }
}

---

Alerting

Принципы алертинга

1. Alert Fatigue Prevention

• Только важные алерты
• Группировка похожих алертов
• Эскалация по времени

2. Alert Classification

• Critical — система недоступна
• Warning — деградация производительности
• Info — информационные уведомления

Правила оповещений Prometheus

# alerts.yml
groups:
- name: application
  rules:
  - alert: HighErrorRate
    expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "High error rate detected"
      description: "Error rate is {{ $value }} for {{ $labels.instance }}"
  
  - alert: HighResponseTime
    expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High response time"
      description: "95th percentile response time is {{ $value }}s"

Конфигурация Alertmanager

# alertmanager.yml
global:
  smtp_smarthost: 'localhost:587'
  smtp_from: 'alerts@example.com'

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'web.hook'

receivers:
- name: 'web.hook'
  webhook_configs:
  - url: 'http://alert-webhook:5001/alerts'
  
- name: 'email'
  email_configs:
  - to: 'admin@example.com'
    subject: 'Alert: {{ .GroupLabels.alertname }}'
    body: |
      {{ range .Alerts }}
      Alert: {{ .Annotations.summary }}
      Description: {{ .Annotations.description }}
      {{ end }}

---

SLI/SLO/SLA

SLI (Индикатор уровня сервиса)

SLI — метрика, которая измеряет качество сервиса.

Примеры SLI:

• Availability: successful_requests / total_requests
• Latency: requests_completed_within_threshold / total_requests
• Throughput: successful_requests_per_second

@Component
public class SLICalculator {
    public double calculateAvailability(String service, Duration window) {
        Counter successfulRequests = Counter.builder("requests.successful")
            .tag("service", service)
            .register(meterRegistry);
            
        Counter totalRequests = Counter.builder("requests.total")
            .tag("service", service)
            .register(meterRegistry);
            
        double successful = successfulRequests.count();
        double total = totalRequests.count();
        
        return total > 0 ? successful / total : 0.0;
    }
    
    public double calculateLatency(String service, Duration threshold) {
        Timer requestTimer = Timer.builder("request.duration")
            .tag("service", service)
            .register(meterRegistry);
            
        // логика расчета латентности
        return 0.95; // пример
    }
}

SLO (Цель уровня сервиса)

SLO — целевое значение для SLI.

Примеры SLO:

• Availability: 99.9% uptime
• Latency: 95% of requests < 200ms
• Error Rate: < 0.1% errors

@Component
public class SLOMonitor {
    private final Map<String, SLO> slos = Map.of(
        "availability", new SLO("availability", 0.999, "99.9% uptime"),
        "latency", new SLO("latency", 0.95, "95% < 200ms"),
        "error_rate", new SLO("error_rate", 0.001, "< 0.1% errors")
    );
    
    public boolean checkSLO(String sloName, double currentValue) {
        SLO slo = slos.get(sloName);
        return currentValue >= slo.getTarget();
    }
}

SLA (Соглашение об уровне сервиса)

SLA — соглашение с клиентами о качестве сервиса.

Компоненты SLA:

• Service Description — описание сервиса
• Performance Metrics — метрики производительности
• Availability Targets — цели доступности
• Remediation — меры по устранению нарушений

---

Практические задания

Задание 1: Prometheus + Grafana

1. Настройте Prometheus для сбора метрик
2. Создайте дашборд в Grafana
3. Настройте алерты

Задание 2: Distributed Tracing

1. Интегрируйте OpenTelemetry в приложение
2. Настройте Jaeger для анализа трассировок
3. Создайте анализ производительности

Задание 3: ELK Stack

1. Настройте централизованное логирование
2. Создайте индексы в Elasticsearch
3. Настройте дашборды в Kibana

---

Дополнительные ресурсы

Инструменты

• Metrics: Prometheus, InfluxDB, TimescaleDB
• Logging: ELK Stack, Fluentd, Splunk
• Tracing: Jaeger, Zipkin, OpenTelemetry
• Dashboards: Grafana, Kibana, DataDog

Книги

• "Observability Engineering" by Charity Majors
• "Site Reliability Engineering" by Google
• "Distributed Systems Observability" by Cindy Sridharan

---

Контрольные вопросы

1. Что такое мониторинг и зачем он нужен?
2. Какие типы метрик вы знаете?
3. Как работает Prometheus и PromQL?
4. Что такое structured logging?
5. Как работает distributed tracing?
6. Что такое три столпа observability?
7. Как настроить алертинг?
8. Что такое SLI/SLO/SLA?