Resilience patterns

Resilience patterns - bu microservice tizimda bir service ishdan chiqsa yoki sekinlashsa, butun system qulab tushmasligi uchun ishlatiladigan patternlar. Bu mavzu Spring Cloud ecosystem’dan keyingi muhim blok sifatida keladi: Resilience4j, fallback strategies, timeout propagation, idempotency keys, graceful degradation.

1. Muammo: microservice’da xato zanjir bo‘lib tarqaladi

Oddiy flow:

Client
  ↓
Gateway
  ↓
Order Service
  ↓
Payment Service
  ↓
Bank API

Agar Bank API sekinlashsa:

Bank API sekin
  ↓
Payment Service kutadi
  ↓
Order Service kutadi
  ↓
Gateway kutadi
  ↓
Client timeout oladi

Yomon tomoni: bitta tashqi service muammosi butun tizimni sekinlashtiradi.

Resilience patternlar shuni oldini oladi.

2. Resilience4j nima?

Resilience4j - Java/Spring Boot’da fault-tolerance uchun ishlatiladigan kutubxona.

U quyidagi patternlarni beradi:

Dependency:

<dependency>
    <groupId>io.github.resilience4j</groupId>
    <artifactId>resilience4j-spring-boot3</artifactId>
</dependency>

3. CircuitBreaker

Nima qiladi?

CircuitBreaker - elektr avtomatga o‘xshaydi.

Agar service ko‘p xato qaytarsa, CircuitBreaker uni vaqtincha "o‘chiradi".

Normal holat:
Order Service → Payment Service

Payment Service xato bera boshlaydi:
Order Service → CircuitBreaker → requestni bloklaydi

Birozdan keyin:
CircuitBreaker test request yuboradi

CircuitBreaker holatlari

CLOSED → normal ishlaydi
OPEN → request yubormaydi
HALF_OPEN → test uchun ozgina request yuboradi

1. CLOSED

Hammasi normal.

Order Service → Payment Service

2. OPEN

Xato ko‘paydi. CircuitBreaker request yubormaydi.

Order Service → fallback response

3. HALF_OPEN

Biroz vaqt o‘tgach, service tuzaldimi deb tekshiradi.

Order Service → Payment Service test call

Agar muvaffaqiyatli bo‘lsa:

HALF_OPEN → CLOSED

Agar yana xato bo‘lsa:

HALF_OPEN → OPEN

Spring Boot misol

@Service
@RequiredArgsConstructor
public class PaymentService {

    private final BankClient bankClient;

    @CircuitBreaker(name = "bankService", fallbackMethod = "paymentFallback")
    public PaymentResponse pay(PaymentRequest request) {
        return bankClient.charge(request);
    }

    public PaymentResponse paymentFallback(PaymentRequest request, Throwable ex) {
        return new PaymentResponse(
                "PENDING",
                "Bank service vaqtincha ishlamayapti. To‘lov keyinroq qayta tekshiriladi."
        );
    }
}

Bu yerda:

@CircuitBreaker(name = "bankService", fallbackMethod = "paymentFallback")

shuni bildiradi:

Agar bankClient.charge() ko‘p xato bersa, fallback ishlaydi.

Config misol

resilience4j:
  circuitbreaker:
    instances:
      bankService:
        slidingWindowSize: 10
        failureRateThreshold: 50
        waitDurationInOpenState: 10s
        permittedNumberOfCallsInHalfOpenState: 3

Tushuntirish:

4. Retry

Nima qiladi?

Retry - vaqtinchalik xatoda qayta urinadi.

Masalan:

1-call → timeout
2-call → timeout
3-call → success

Bu foydali bo‘lishi mumkin, lekin noto‘g‘ri ishlatilsa tizimni yanada bosib yuboradi.

Qachon Retry qilish mumkin?

Retry qilish mumkin:

Retry misol

@Service
@RequiredArgsConstructor
public class NotificationService {

    private final SmsClient smsClient;

    @Retry(name = "smsService", fallbackMethod = "smsFallback")
    public SmsResponse sendSms(SmsRequest request) {
        return smsClient.send(request);
    }

    public SmsResponse smsFallback(SmsRequest request, Throwable ex) {
        return new SmsResponse("FAILED", "SMS hozir yuborilmadi");
    }
}

Config:

resilience4j:
  retry:
    instances:
      smsService:
        maxAttempts: 3
        waitDuration: 500ms

Bu nima qiladi?

1-urinish
500ms kutadi
2-urinish
500ms kutadi
3-urinish
xato bo‘lsa fallback

Retry xavfi

Eng katta xato:

1000 ta request
har biri 3 marta retry qiladi
=
3000 ta request

Bu sekinlashgan service’ni yanada yiqitadi.

Shuning uchun retry doim quyidagilar bilan birga o‘ylanadi:

Retry + Timeout + CircuitBreaker + Idempotency

5. RateLimiter

Nima qiladi?

RateLimiter - ma’lum vaqt ichida nechta request o‘tishini cheklaydi.

Masalan:

1 sekundda faqat 100 ta request

Agar undan ko‘p bo‘lsa:

Too many requests

Qayerda ishlatiladi?

Misol

@Service
@RequiredArgsConstructor
public class ReportService {

    private final ReportClient reportClient;

    @RateLimiter(name = "reportLimiter", fallbackMethod = "rateLimitFallback")
    public ReportResponse generateReport(Long userId) {
        return reportClient.generate(userId);
    }

    public ReportResponse rateLimitFallback(Long userId, Throwable ex) {
        return new ReportResponse("TOO_MANY_REQUESTS", "Birozdan keyin urinib ko‘ring");
    }
}

Config:

resilience4j:
  ratelimiter:
    instances:
      reportLimiter:
        limitForPeriod: 10
        limitRefreshPeriod: 1s
        timeoutDuration: 0

Bu nimani bildiradi?

1 sekundda 10 ta request o‘tadi.
11-request darhol reject bo‘ladi.

6. Bulkhead

Nima qiladi?

Bulkhead - tizim resurslarini bo‘lib qo‘yadi.

Kema misoli:

Kemaning bitta qismi suvga to‘lsa ham, butun kema cho‘kmaydi.

Microservice’da ham shunday:

Payment API sekinlashsa,
Notification API uchun threadlar band bo‘lib qolmasligi kerak.

Bulkhead bo‘lmasa

Thread pool: 100 thread

Payment call sekinlashdi
↓
100 thread ham Payment’da kutib qoldi
↓
Order, User, Notification ham ishlamay qoldi

Bulkhead bilan

Payment uchun: 20 thread
Notification uchun: 20 thread
User uchun: 20 thread

Payment sekinlashsa ham,
boshqa qismlar ishlashda davom etadi.

Misol

@Service
@RequiredArgsConstructor
public class PaymentClientService {

    private final PaymentClient paymentClient;

    @Bulkhead(name = "paymentBulkhead", fallbackMethod = "bulkheadFallback")
    public PaymentResponse pay(PaymentRequest request) {
        return paymentClient.pay(request);
    }

    public PaymentResponse bulkheadFallback(PaymentRequest request, Throwable ex) {
        return new PaymentResponse("BUSY", "Payment service hozir band");
    }
}

Config:

resilience4j:
  bulkhead:
    instances:
      paymentBulkhead:
        maxConcurrentCalls: 20
        maxWaitDuration: 100ms

Bu nimani bildiradi?

7. TimeLimiter / Timeout

Nima qiladi?

Timeout - call juda uzoq cho‘zilib ketsa, uni to‘xtatadi.

Yomon holat:

Order Service → Payment Service
              kutadi...
              kutadi...
              kutadi...

Yaxshi holat:

Order Service → Payment Service
              2 sekund kutadi
              timeout
              fallback

Timeout nega muhim?

Timeout bo‘lmasa:

• Threadlar band bo‘lib qoladi.

• Connection pool to‘lib qoladi.

• Gateway timeout beradi.

• Client tajribasi yomonlashadi.

• System domino kabi yiqiladi.

Timeout propagation

Senior darajada eng muhim joy shu.

Agar client umumiy 5 sekund kutsa, ichki service’lar ham shunga mos timeout olishi kerak.

Yomon:

Client timeout: 5s
Gateway timeout: 10s
Order timeout: 15s
Payment timeout: 30s

Bu xato. Chunki client allaqachon ketib bo‘lgan bo‘ladi, lekin backend hali ham ishlaydi.

Yaxshi:

Client timeout: 5s
Gateway timeout: 4.5s
Order timeout: 3s
Payment timeout: 1.5s
Bank API timeout: 1s

Ya’ni timeout ichkariga kirgan sari kichrayadi.

8. Fallback strategies

Fallback nima?

Fallback - asosiy service ishlamasa, alternativ javob qaytarish.

Masalan:

Recommendation Service ishlamadi
↓
Popular products qaytar

Yoki:

Payment Service timeout
↓
Order status = PENDING_PAYMENT

Fallback turlari

Real misollar

1. Product rating fallback

Rating Service ishlamasa:
rating = null

Response:

{
  "id": 10,
  "name": "iPhone 15",
  "rating": null
}

Bu yaxshi. Product sahifasi butunlay yiqilmaydi.

2. Payment fallback

Payment’da ehtiyot bo‘lish kerak.

Yomon:

Payment timeout bo‘ldi
↓
Order SUCCESS qilib qo‘yish

Bu xavfli.

Yaxshi:

Payment timeout bo‘ldi
↓
Order PENDING_PAYMENT
↓
Payment status keyin tekshiriladi

3. Notification fallback

SMS yuborilmadi
↓
Email yubor

Yoki:

SMS yuborilmadi
↓
notification_outbox table’ga yoz
↓
background job keyin yuboradi

9. Idempotency keys

Muammo

User payment tugmasini ikki marta bosdi:

POST /payments
POST /payments

Yoki retry sababli request ikki marta ketdi:

1-request timeout
2-request retry

Agar idempotency bo‘lmasa:

Userdan 2 marta pul yechilishi mumkin.

Idempotency nima?

Idempotency - bir xil request bir necha marta yuborilsa ham natija bir marta bajariladi.

Client header yuboradi:

Idempotency-Key: 8f7b4c1a-123-payment

Backend bu key’ni saqlaydi.

Idempotency-Key bor
  ↓
oldin ishlanganmi?
  ↓
ha → eski response qaytar
yo‘q → operation bajar va key’ni saqla

Payment misol

@PostMapping("/payments")
public PaymentResponse pay(
        @RequestHeader("Idempotency-Key") String idempotencyKey,
        @RequestBody PaymentRequest request
) {
    return paymentService.pay(idempotencyKey, request);
}

Service:

@Transactional
public PaymentResponse pay(String key, PaymentRequest request) {
    Optional<PaymentLog> existing = paymentLogRepository.findByIdempotencyKey(key);

    if (existing.isPresent()) {
        return existing.get().toResponse();
    }

    PaymentResponse response = bankClient.charge(request);

    paymentLogRepository.save(
            new PaymentLog(key, request.amount(), response.status())
    );

    return response;
}

Database’da unique constraint kerak:

ALTER TABLE payment_log
ADD CONSTRAINT uk_payment_idempotency_key UNIQUE (idempotency_key);

Bu juda muhim. Faqat Java check yetmaydi. Parallel request kelganda DB unique constraint himoya qiladi.

10. Graceful degradation

Nima?

Graceful degradation - tizim to‘liq yiqilmasdan, kamroq imkoniyat bilan ishlashda davom etishi.

Masalan:

Search service ishlamayapti
↓
Category bo‘yicha oddiy list qaytar

Yoki:

Recommendation ishlamayapti
↓
Popular products ko‘rsat

Yoki:

Analytics ishlamayapti
↓
User flow davom etadi, analytics keyin yuboriladi

Muhim qoida

Har bir service bir xil critical emas.

11. Patternlarni noto‘g‘ri ishlatish

Xato 1: Har joyga Retry qo‘yish

Yomon:

Gateway retry
Order retry
Payment retry
Bank client retry

Natija:

1 request → 3 × 3 × 3 × 3 = 81 request

Bu retry storm deyiladi.

Xato 2: Fallback’da yolg‘on success qaytarish

Yomon:

public PaymentResponse fallback(...) {
    return new PaymentResponse("SUCCESS");
}

Bu production’da moliyaviy muammo keltiradi.

To‘g‘ri:

public PaymentResponse fallback(...) {
    return new PaymentResponse("PENDING");
}

Xato 3: Timeout yo‘q

Timeoutsiz HTTP call - production’da xavfli.

Har bir external call’da timeout bo‘lishi kerak:

connect timeout
read timeout
overall timeout

Xato 4: CircuitBreaker threshold juda agressiv

Masalan:

failureRateThreshold: 10
slidingWindowSize: 5

5 ta requestdan bittasi xato bo‘lsa, breaker ochilishi mumkin. Bu false alarm beradi.

12. Real architecture example

Order yaratish flow:

Client
  ↓
Gateway
  ↓
Order Service
  ├── Inventory Service
  ├── Payment Service
  └── Notification Service

Resilience bilan:

Order Service
  ├── Inventory Service
  │     ├── Timeout: 1s
  │     ├── Retry: 2 attempts
  │     └── CircuitBreaker
  │
  ├── Payment Service
  │     ├── Timeout: 2s
  │     ├── Idempotency-Key
  │     ├── CircuitBreaker
  │     └── Fallback: PENDING_PAYMENT
  │
  └── Notification Service
        ├── Async
        ├── Retry
        └── Fallback: Outbox table

13. Senior darajadagi qarorlar

Senior developer faqat annotation qo‘ymaydi. U quyidagi qarorlarni beradi:

1. Qaysi xatoda retry qilamiz?

Timeout → retry mumkin
500 → retry mumkin
400 → retry yo‘q
401 → retry yo‘q
Payment duplicate risk → retry faqat idempotency bilan

2. Timeout qancha bo‘ladi?

Har bir service uchun alohida:

Payment: 2s
Inventory: 1s
Notification: async
Recommendation: 500ms

3. Fallback business jihatdan to‘g‘rimi?

Technical fallback yetarli emas. Business holat to‘g‘ri bo‘lishi kerak.

Payment failed → order success emas
Payment unknown → pending
Notification failed → order failed emas

4. Observability bormi?

Resilience pattern ishlayotganini ko‘rish kerak:

circuitbreaker.open.count
retry.calls
fallback.count
timeout.count
rate_limited.count

14. Minimal production config namunasi

resilience4j:
  circuitbreaker:
    instances:
      paymentService:
        slidingWindowType: COUNT_BASED
        slidingWindowSize: 20
        minimumNumberOfCalls: 10
        failureRateThreshold: 50
        waitDurationInOpenState: 15s
        permittedNumberOfCallsInHalfOpenState: 5

  retry:
    instances:
      paymentService:
        maxAttempts: 2
        waitDuration: 300ms

  bulkhead:
    instances:
      paymentService:
        maxConcurrentCalls: 30
        maxWaitDuration: 100ms

  ratelimiter:
    instances:
      paymentService:
        limitForPeriod: 100
        limitRefreshPeriod: 1s
        timeoutDuration: 0

15. Qisqa interview javob

Agar suhbatda so‘rasa:

Resilience patterns nima?

Javob:

Resilience patterns microservice tizimda xatolar tarqalib ketmasligi uchun ishlatiladi.
Masalan, CircuitBreaker buzilgan service’ga request yuborishni vaqtincha to‘xtatadi,
Retry vaqtinchalik xatoda qayta urinadi, RateLimiter ortiqcha requestni cheklaydi,
Bulkhead resurslarni izolyatsiya qiladi, Timeout sekin call’larni to‘xtatadi,
Fallback esa service ishlamasa alternativ javob qaytaradi.

Senior darajada bularni biznes holat bilan moslashtirish muhim:
payment’da fallback SUCCESS bo‘lmasligi kerak, PENDING bo‘lishi kerak.
Retry esa faqat idempotency bilan xavfsiz bo‘ladi.

16. Eng muhim xulosa

Resilience patternlarning asosiy maqsadi:

Bitta service yiqilsa, butun system yiqilmasin.

Eng kerakli kombinatsiya:

Timeout
+ CircuitBreaker
+ Retry
+ Fallback
+ Bulkhead
+ Idempotency
+ Observability

Senior developer uchun eng muhim narsa - annotation yodlash emas, qaysi biznes holatda qaysi pattern xavfsiz yoki xavfli ekanini bilish.