Observability & Actuator

Bu bo‘limda Spring Boot Actuator endpoints, custom health indicators, Micrometer metrics, Prometheus + Grafana, distributed tracing, va structured logging with MDCni ko‘ramiz.

Spring Boot Actuator productionga chiqarilgan applicationni monitoring va management qilish uchun endpointlar, health checklar, metrics va boshqa imkoniyatlar beradi. (Home)

1. Observability nima?

Observability - application ichida nima bo‘layotganini tashqaridan tushuna olish.

Oddiy monitoring:

Server ishlayaptimi?
CPU qancha?
RAM qancha?

Observability esa chuqurroq:

Qaysi endpoint sekin?
Qaysi service xato beryapti?
Qaysi request qayerda to‘xtab qolyapti?
DB query sekinmi?
Kafka consumer ortda qolyaptimi?
Redis ishlayaptimi?

Production backendda 3 ta asosiy ustun bor:

2. Actuator nima?

Spring Boot Actuator - application haqida production ma’lumotlarini endpointlar orqali chiqaradi.

Dependency:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

Shundan keyin endpointlar paydo bo‘ladi:

/actuator/health
/actuator/info
/actuator/metrics
/actuator/prometheus
/actuator/env
/actuator/loggers

Default holatda hamma endpointlar web orqali ochiq bo‘lmasligi mumkin. Xavfsizlik uchun Spring Boot faqat keraklilarini expose qiladi.

3. Asosiy Actuator endpointlar

Productionda hammasini ochib yuborish xavfli.

Yomon:

management:
  endpoints:
    web:
      exposure:
        include: "*"

Yaxshiroq:

management:
  endpoints:
    web:
      exposure:
        include: health,info,metrics,prometheus,loggers

4. Health endpoint

Eng ko‘p ishlatiladigan endpoint:

GET /actuator/health

Javob:

{
  "status": "UP"
}

Agar DB, Redis yoki boshqa dependencyda muammo bo‘lsa:

{
  "status": "DOWN"
}

Spring Boot Actuator health, metrics va monitoring uchun production-ready imkoniyatlar beradi. (Home)

5. Health details ko‘rsatish

Developmentda:

management:
  endpoint:
    health:
      show-details: always

Javob:

{
  "status": "UP",
  "components": {
    "db": {
      "status": "UP"
    },
    "redis": {
      "status": "UP"
    },
    "diskSpace": {
      "status": "UP"
    }
  }
}

Productionda ehtiyot bo‘lish kerak:

management:
  endpoint:
    health:
      show-details: when-authorized

Chunki health details ichida infrastructure haqida sezgir ma’lumot chiqishi mumkin.

6. Kubernetes uchun liveness va readiness

Kubernetesda odatda ikki xil health kerak:

Config:

management:
  endpoint:
    health:
      probes:
        enabled: true

Endpointlar:

/actuator/health/liveness
/actuator/health/readiness

Tushuncha:

Liveness DOWN  -> Kubernetes appni restart qiladi
Readiness DOWN -> Kubernetes appga traffic yubormaydi

Masalan DB vaqtincha down bo‘lsa, appni darrov restart qilish shart emas. Ba’zan readiness DOWN bo‘lishi yetarli.

7. Custom HealthIndicator

Ba’zan o‘zimizning dependency healthini tekshirish kerak.

Masalan external payment service:

@Component
public class PaymentServiceHealthIndicator implements HealthIndicator {

    private final PaymentClient paymentClient;

    public PaymentServiceHealthIndicator(PaymentClient paymentClient) {
        this.paymentClient = paymentClient;
    }

    @Override
    public Health health() {
        try {
            boolean available = paymentClient.ping();

            if (available) {
                return Health.up()
                        .withDetail("paymentService", "Available")
                        .build();
            }

            return Health.down()
                    .withDetail("paymentService", "Unavailable")
                    .build();

        } catch (Exception e) {
            return Health.down(e)
                    .withDetail("paymentService", "Error")
                    .build();
        }
    }
}

Endi /actuator/health ichida custom component chiqadi.

8. Custom healthda og‘ir ish qilmang

Yomon:

@Override
public Health health() {
    // 5 sekundlik external API call
    // katta DB query
    // ko‘p table scan
}

Health endpoint tez ishlashi kerak. Aks holda monitoring systemning o‘zi appga yuk bo‘ladi.

Yaxshi:

- qisqa timeout
- lightweight ping
- cache qilingan status
- circuit breaker bilan himoyalangan check

9. Info endpoint

/actuator/info orqali app haqida ma’lumot chiqarish mumkin.

Config:

management:
  info:
    env:
      enabled: true

info:
  app:
    name: order-service
    version: 1.0.0
    description: Order management service

Javob:

{
  "app": {
    "name": "order-service",
    "version": "1.0.0",
    "description": "Order management service"
  }
}

Foydasi:

Qaysi version deploy bo‘lgan?
Qaysi service ishlayapti?
Build info bormi?

10. Metrics nima?

Metrics - raqamli o‘lchovlar.

Masalan:

HTTP request soni
HTTP response vaqti
Error count
JVM memory
Thread count
DB connection pool
Cache hit/miss
Kafka consumer lag

Spring Boot Actuator Micrometer uchun dependency management va auto-configuration beradi. Micrometer Prometheus, Datadog, New Relic, OTLP, Graphite kabi ko‘p monitoring systemlarni qo‘llab-quvvatlaydi. (Home)

11. Built-in metrics

Actuator ko‘p metriclarni avtomatik chiqaradi:

Endpoint:

GET /actuator/metrics

Aniq metric:

GET /actuator/metrics/http.server.requests

12. Micrometer nima?

Micrometer - metrics facade.

SLF4J logging uchun qanday abstraction bo‘lsa, Micrometer metrics uchun shunday abstraction.

Kodda siz Micrometer bilan ishlaysiz:

Counter counter = meterRegistry.counter("orders.created");
counter.increment();

Keyin metrics qayerga ketishini config hal qiladi:

Prometheus
Datadog
New Relic
OTLP
Graphite
...

Micrometer o‘zini JVM observability uchun vendor-neutral facade sifatida ta’riflaydi. (Micrometer Application Observability)

13. Custom Counter

Masalan order yaratilganini sanash:

@Service
public class OrderService {

    private final OrderRepository orderRepository;
    private final Counter ordersCreatedCounter;

    public OrderService(
            OrderRepository orderRepository,
            MeterRegistry meterRegistry
    ) {
        this.orderRepository = orderRepository;
        this.ordersCreatedCounter = Counter.builder("orders.created")
                .description("Total created orders")
                .register(meterRegistry);
    }

    @Transactional
    public OrderDto createOrder(CreateOrderRequest request) {
        Order order = orderRepository.save(new Order(request.userId()));

        ordersCreatedCounter.increment();

        return OrderDto.from(order);
    }
}

Metric nomi:

orders.created

Prometheusda odatda:

orders_created_total

ko‘rinishida chiqadi.

14. Custom Timer

Method qancha vaqt olganini o‘lchash:

@Service
public class PaymentService {

    private final PaymentClient paymentClient;
    private final Timer paymentTimer;

    public PaymentService(
            PaymentClient paymentClient,
            MeterRegistry meterRegistry
    ) {
        this.paymentClient = paymentClient;
        this.paymentTimer = Timer.builder("payment.charge.duration")
                .description("Payment charge duration")
                .register(meterRegistry);
    }

    public PaymentResult charge(PaymentRequest request) {
        return paymentTimer.record(() -> paymentClient.charge(request));
    }
}

Timer yordamida:

count
total time
max
percentiles/histogram

kuzatiladi.

15. Gauge

Gauge - hozirgi qiymatni o‘lchaydi.

Masalan queue size:

@Component
public class QueueMetrics {

    private final BlockingQueue<Job> queue = new LinkedBlockingQueue<>();

    public QueueMetrics(MeterRegistry meterRegistry) {
        Gauge.builder("jobs.queue.size", queue, BlockingQueue::size)
                .description("Current job queue size")
                .register(meterRegistry);
    }
}

Counter va Gauge farqi:

16. Metric taglari

Taglar metricni dimensionlarga ajratadi.

Counter.builder("orders.created")
        .tag("status", "success")
        .tag("paymentType", "card")
        .register(meterRegistry)
        .increment();

Prometheusda:

orders_created_total{status="success", paymentType="card"}

Lekin taglarda ehtiyot bo‘lish kerak.

Yomon:

.tag("userId", userId.toString())
.tag("orderId", orderId.toString())

Bu high cardinality muammo beradi.

Yomon taglar:

userId
orderId
phone
email
uuid
requestId

Yaxshi taglar:

status
method
endpoint
exception
region
paymentType

17. Prometheus endpoint

Prometheus uchun dependency:

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

Config:

management:
  endpoints:
    web:
      exposure:
        include: health,info,metrics,prometheus

Endpoint:

GET /actuator/prometheus

Micrometer docs’da Prometheus Spring Boot Actuator mavjud bo‘lsa, Prometheus actuator endpoint avtomatik sozlanishini aytadi. (Micrometer Documentation)

18. Prometheus qanday ishlaydi?

Prometheus applicationga metric yubormaydi. Aksincha, Prometheus applicationdan metriclarni scrape qiladi.

Spring Boot app exposes /actuator/prometheus
Prometheus periodically scrapes it
Grafana Prometheusdan data o‘qib dashboard chizadi

Oqim:

Spring Boot
  ↓ /actuator/prometheus
Prometheus
  ↓ query
Grafana

19. Prometheus scrape config

prometheus.yml:

scrape_configs:
  - job_name: 'order-service'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['order-service:8080']

Agar local test bo‘lsa:

targets: ['localhost:8080']

Kubernetesda esa odatda ServiceMonitor/PodMonitor ishlatiladi.

20. Grafana nima qiladi?

Grafana Prometheusdagi metriclarni dashboard qilib ko‘rsatadi.

Masalan dashboard:

- HTTP request rate
- P95 latency
- Error rate
- JVM memory
- DB connection pool usage
- CPU
- Kafka lag
- Cache hit ratio

Grafana alert ham qo‘yishi mumkin:

P95 latency > 1s for 5 minutes
Error rate > 5%
DB connections active > 90%

21. Muhim production metriclar

Technical metrics yetarli emas. Business metrics ham kerak.

Masalan:

payments.failed.count
orders.created.count
sms.delivery.failed

22. RED method

Service monitoring uchun oddiy model:

HTTP API uchun:

Rate    -> requests per second
Errors  -> 5xx / 4xx
Duration -> p50 / p95 / p99 latency

Bu backend service uchun eng kerakli dashboardlardan biri.

23. USE method

Infrastructure/resource monitoring uchun:

Masalan DB connection pool:

Utilization -> active connections / max connections
Saturation  -> pending threads
Errors      -> connection timeout

24. Distributed tracing nima?

Microservice architecture’da bitta request bir nechta servicelardan o‘tadi:

API Gateway
  ↓
Order Service
  ↓
Payment Service
  ↓
Notification Service

Agar request sekin bo‘lsa, qaysi service aybdor?

Distributed tracing shuni ko‘rsatadi.

Trace:

traceId = bitta requestning umumiy IDsi
spanId = har bir service/operation qismi

Misol:

traceId: abc123
  span: API Gateway 20ms
  span: Order Service 80ms
  span: Payment Service 900ms
  span: Notification Service 30ms

Bu yerda Payment Service sekinligi ko‘rinadi.

Spring Boot Actuator Micrometer Tracing uchun auto-configuration va dependency management beradi. (Home)

25. Micrometer Tracing

Spring Boot 3 davrida tracing uchun asosiy yo‘l - Micrometer Tracing.

Micrometer Tracing popular tracer kutubxonalari uchun facade beradi va kodni vendor lock-in’dan himoya qilishga yordam beradi. (Micrometer Documentation)

Dependency misoli:

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>

Config:

management:
  tracing:
    sampling:
      probability: 1.0

1.0 - hamma request trace qilinadi.

Productionda ko‘pincha:

management:
  tracing:
    sampling:
      probability: 0.1

ya’ni 10% sampling ishlatiladi.

26. Zipkin bilan tracing

Zipkin ishlatilsa:

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-brave</artifactId>
</dependency>

<dependency>
    <groupId>io.zipkin.reporter2</groupId>
    <artifactId>zipkin-reporter-brave</artifactId>
</dependency>

Config:

management:
  tracing:
    sampling:
      probability: 1.0

spring:
  zipkin:
    base-url: http://localhost:9411

Eslatma: Spring Boot versiyasiga qarab tracing/exporter config nomlari farq qilishi mumkin. Yangi loyihalarda OpenTelemetry/OTLP yo‘nalishi ko‘proq mos keladi.

27. Structured logging nima?

Oddiy log:

Order created successfully

Bu kam ma’lumot beradi.

Structured log:

{
  "level": "INFO",
  "service": "order-service",
  "traceId": "abc123",
  "spanId": "def456",
  "userId": 10,
  "orderId": 55,
  "message": "Order created"
}

Buni ELK, Loki, Datadog kabi systemlarda qidirish oson.

28. MDC nima?

MDC - Mapped Diagnostic Context.

Loglarga avtomatik context qo‘shish uchun ishlatiladi.

Masalan har requestga requestId qo‘shamiz.

Filter:

@Component
public class RequestIdFilter extends OncePerRequestFilter {

    private static final String REQUEST_ID = "requestId";

    @Override
    protected void doFilterInternal(
            HttpServletRequest request,
            HttpServletResponse response,
            FilterChain filterChain
    ) throws ServletException, IOException {

        String requestId = Optional
                .ofNullable(request.getHeader("X-Request-Id"))
                .orElse(UUID.randomUUID().toString());

        MDC.put(REQUEST_ID, requestId);
        response.setHeader("X-Request-Id", requestId);

        try {
            filterChain.doFilter(request, response);
        } finally {
            MDC.remove(REQUEST_ID);
        }
    }
}

Logback pattern:

<pattern>
    %d{yyyy-MM-dd HH:mm:ss} [%thread] %-5level traceId=%X{traceId} requestId=%X{requestId} %logger{36} - %msg%n
</pattern>

29. MDC’da finally shart

Yomon:

MDC.put("requestId", requestId);
filterChain.doFilter(request, response);

Agar thread pool ishlatilsa, context keyingi requestga "oqib" ketishi mumkin.

To‘g‘ri:

try {
    filterChain.doFilter(request, response);
} finally {
    MDC.clear();
}

Yoki aniq keylarni remove qilish:

finally {
    MDC.remove("requestId");
}

30. Trace ID loglarda

Tracing yoqilganda loglarga traceId va spanId qo‘shish juda foydali.

Log:

2026-05-21 INFO traceId=abc123 spanId=def456 Order created

Keyin siz:

traceId=abc123

bo‘yicha hamma servicelardagi loglarni topasiz.

Bu microservice debuggingda juda katta yordam beradi.

31. Log levelni runtime o‘zgartirish

Actuator /actuator/loggers endpointi orqali log levelni runtime ko‘rish va o‘zgartirish mumkin.

Masalan:

GET /actuator/loggers/com.example.order

POST:

{
  "configuredLevel": "DEBUG"
}

Bu productionda vaqtincha debug qilish uchun foydali.

Lekin endpointni himoyalash shart.

32. Actuator security

Actuator endpointlar xavfli bo‘lishi mumkin.

Masalan:

/env
/beans
/heapdump
/threaddump
/loggers

Bular ichki ma’lumotlarni chiqarishi mumkin.

Security config:

@Bean
public SecurityFilterChain actuatorSecurity(HttpSecurity http)
        throws Exception {

    return http
            .securityMatcher(EndpointRequest.toAnyEndpoint())
            .authorizeHttpRequests(auth -> auth
                    .requestMatchers(EndpointRequest.to("health", "info"))
                    .permitAll()
                    .anyRequest()
                    .hasRole("ACTUATOR_ADMIN")
            )
            .httpBasic(Customizer.withDefaults())
            .build();
}

Application API uchun alohida filter chain yozish mumkin.

33. Health endpointni public qilish

Kubernetes yoki load balancer health check uchun /actuator/health ochiq bo‘lishi mumkin.

Lekin:

/actuator/env
/actuator/heapdump
/actuator/loggers

public bo‘lmasligi kerak.

Yaxshi amaliyot:

health/info -> public yoki internal
metrics/prometheus -> faqat monitoring network
env/beans/heapdump/loggers -> auth required yoki disabled

34. Custom Actuator endpoint

Ba’zan custom management endpoint kerak bo‘ladi.

@Component
@Endpoint(id = "feature-flags")
public class FeatureFlagsEndpoint {

    private final FeatureFlagService featureFlagService;

    public FeatureFlagsEndpoint(FeatureFlagService featureFlagService) {
        this.featureFlagService = featureFlagService;
    }

    @ReadOperation
    public Map<String, Boolean> flags() {
        return featureFlagService.getFlags();
    }
}

Endpoint:

GET /actuator/feature-flags

Configda expose qilish:

management:
  endpoints:
    web:
      exposure:
        include: health,info,feature-flags

Custom endpointlarni ehtiyotkorlik bilan ishlating. Ular production control surface bo‘lib qoladi.

35. Observability va Alerting

Observability faqat dashboard emas. Alert ham kerak.

Yomon alert:

CPU > 80%

Bu har doim ham userga zarar degani emas.

Yaxshi alert:

5xx error rate > 5% for 5 minutes
p95 latency > 1s for 10 minutes
payment failed count increased abnormally
consumer lag increasing for 15 minutes
DB connection pool exhausted

Alert user impactga yaqin bo‘lishi kerak.

36. Real project uchun minimal setup

Dependencylar:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

Config:

management:
  endpoints:
    web:
      exposure:
        include: health,info,metrics,prometheus,loggers
  endpoint:
    health:
      probes:
        enabled: true
      show-details: when-authorized
  metrics:
    tags:
      application: order-service

Bu bilan:

/actuator/health
/actuator/health/liveness
/actuator/health/readiness
/actuator/metrics
/actuator/prometheus

tayyor bo‘ladi.

37. Real dashboardda nimalar bo‘lishi kerak?

Order service uchun:

38. Common mistakes

39. Interview savollar

Savol 1: Observability nima?

Javob:

Observability application ichki holatini logs, metrics va traces orqali tashqaridan tushunish imkonidir. Bu production muammolarini tez topish, latency, error, dependency va business holatlarni kuzatish uchun kerak.

Savol 2: Spring Boot Actuator nima beradi?

Javob:

Actuator production monitoring va management uchun endpointlar beradi: health, info, metrics, prometheus, loggers, env, beans, mappings va boshqalar. U application holatini kuzatish va boshqarishga yordam beradi.

Savol 3: Health va readiness farqi nima?

Javob:

Health umumiy app holatini bildiradi. Kubernetesda liveness app tirikmi yoki restart kerakmi, readiness esa app traffic qabul qilishga tayyormi degan savolga javob beradi.

Savol 4: Micrometer nima?

Javob:

Micrometer metrics facade. Kodda bitta API orqali metrics yozasiz, keyin uni Prometheus, Datadog, New Relic, OTLP kabi monitoring tizimlariga ulash mumkin.

Savol 5: Counter, Gauge, Timer farqi nima?

Javob:

Counter faqat oshadigan qiymat uchun, masalan total orders. Gauge hozirgi holat uchun, masalan queue size. Timer esa duration va countni o‘lchaydi, masalan payment request vaqti.

Savol 6: Distributed tracing nima?

Javob:

Distributed tracing bitta request bir nechta service orqali o‘tganda uning yo‘lini traceId va spanId orqali ko‘rsatadi. Bu qaysi service sekin yoki xato berayotganini topishga yordam beradi.

Savol 7: MDC nima uchun kerak?

Javob:

MDC loglarga requestId, traceId, userId kabi context qo‘shish uchun ishlatiladi. Bu loglarni qidirish va bir request bo‘yicha bog‘lashni osonlashtiradi. Thread pool sababli MDC tozalanishi shart.

40. Qisqa xulosa

Eng muhim fikr:

Observability productionda "nima bo‘ldi?" degan savolga tez va aniq javob berish uchun kerak. Logs, metrics va traces birga bo‘lmasa, muammoni topish taxmin qilishga aylanadi.