Taqsimlangan tizimlar asoslari

Distributed systems fundamentals ichiga quyidagilar kiradi: CAP theorem, consistency models, idempotency, at-least-once delivery, distributed transactions, SAGA, 2PC, circuit breaker, backpressure, rate limiting.

1. Distributed system nima?

Distributed system — bitta katta monolit emas, bir nechta alohida servislar birga ishlaydigan tizim.

Masalan:

Client
  ↓
API Gateway
  ↓
Order Service
  ↓
Payment Service
  ↓
Inventory Service
  ↓
Notification Service

Har bir servis alohida:

• o‘z database’iga ega bo‘lishi mumkin;

• alohida deploy qilinadi;

• alohida scale qilinadi;

• alohida xato qilishi mumkin.

Oddiy ko‘rinadi, lekin katta muammo shuki:

Distributed system’da hamma narsa har doim ishlaydi deb bo‘lmaydi.

Network uzilishi mumkin. Kafka kechikishi mumkin. Payment service timeout berishi mumkin. Database sekinlashishi mumkin. Bir servis yangi data ko‘radi, boshqasi eski data ko‘radi.

Senior developer mana shu muammolarni oldindan hisobga oladi.

2. Monolith vs Distributed system

Monolith

Client → One Backend → One Database

Afzalliklari:

• tushunish oson;

• transaction oson;

• debugging oson;

• deploy oddiy;

• localda yuritish oson.

Kamchiliklari:

• katta bo‘lsa sekin rivojlanadi;

• hamma jamoa bitta codebase’da ishlaydi;

• bitta moduldagi xato butun appga ta’sir qiladi;

• scale qilish nozikroq bo‘ladi.

Distributed system / Microservices

Client → Gateway → Service A → Service B → Service C

Afzalliklari:

• servislarni alohida scale qilish mumkin;

• alohida deploy qilish mumkin;

• jamoalar mustaqil ishlaydi;

• texnologiyani servis bo‘yicha tanlash mumkin.

Kamchiliklari:

• network muammolari ko‘payadi;

• transaction murakkablashadi;

• logging/tracing kerak bo‘ladi;

• data consistency qiyinlashadi;

• monitoring majburiy bo‘ladi.

Muhim qoida:

Microservice — bu faqat @RestControllerni ko‘paytirish emas. Bu operational complexity.

3. CAP theorem

CAP theorem distributed system’dagi eng asosiy tushunchalardan biri.

CAP uchta narsa haqida gapiradi:

CAP aytadi:

Network partition bo‘lganda siz bir vaqtda ham kuchli consistency, ham availability’ni to‘liq saqlay olmaysiz. Tanlov qilasiz.

3.1. Consistency

Consistency degani:

User data yozdi. Keyingi o‘qishda hamma joyda shu yangi data ko‘rinadi.

Masalan:

User balance: 100$
User 20$ yechdi
Yangi balance: 80$

Agar boshqa servis hali ham 100$ ko‘rsa, consistency buzilgan bo‘ladi.

3.2. Availability

Availability degani:

Tizim har qanday holatda javob qaytarishga harakat qiladi.

Masalan, bitta node o‘chsa ham servis response beradi.

Lekin response eski data bilan bo‘lishi mumkin.

3.3. Partition tolerance

Partition degani — network muammosi.

Masalan:

Service A database node 1 bilan gaplasha oladi
Service B database node 2 bilan gaplasha oladi
Lekin node 1 va node 2 bir-biri bilan gaplasha olmaydi

Distributed system’da network ishonchsiz. Shuning uchun partition tolerance deyarli majburiy.

4. CAP real misol

Tasavvur qiling, sizda ecommerce bor.

Order Service
Inventory Service
Payment Service

User product sotib olmoqda.

Inventory’da 1 dona mahsulot qoldi.

Bir vaqtning o‘zida ikki user buyurtma qildi.

Variant 1: Consistency muhim

Tizim shunday deydi:

Men aniq bilmagunimcha sotmayman.

Natija:

• oversell bo‘lmaydi;

• lekin ayrim requestlar rad qilinadi;

• availability pasayadi.

Bu bank, payment, inventory kabi joylarda muhim.

Variant 2: Availability muhim

Tizim shunday deydi:

Hozircha buyurtmani qabul qilaman, keyin tekshiraman.

Natija:

• servis ishlashda davom etadi;

• user tez javob oladi;

• lekin keyin order cancel bo‘lishi mumkin.

Bu booking, marketplace, event-driven order processing’da uchrashi mumkin.

5. Consistency models

Distributed system’da consistency bitta tur emas. Bir nechta modeli bor.

5.1. Strong consistency

Strong consistency — yozilgan data darhol hamma joyda ko‘rinadi.

Misol:

balance = 100
withdraw 20
balance = 80

Keyingi har qanday o‘qishda 80 ko‘rinishi kerak.

Mos joylar:

• bank balance;

• payment;

• inventory reservation;

• account security;

• permission checks.

Kamchiligi:

• sekinroq bo‘lishi mumkin;

• availability kamayishi mumkin;

• distributed lock yoki transaction kerak bo‘lishi mumkin.

5.2. Eventual consistency

Eventual consistency — data darhol emas, birozdan keyin hamma joyda bir xil bo‘ladi.

Masalan:

Order created
 ↓
Kafka event
 ↓
Notification Service updated
 ↓
Analytics Service updated

Order yaratildi, lekin analytics 2 soniyadan keyin yangilandi.

Bu normal.

Mos joylar:

• notification;

• search index;

• analytics;

• recommendation;

• reporting;

• email sending;

• audit log.

Muhim fikr:

Eventual consistency xato emas. To‘g‘ri joyda ishlatilsa, bu yaxshi arxitektura.

6. Idempotency

Idempotency — bir xil request bir necha marta kelsa ham natija bir marta bajarilgandek bo‘lishi.

Bu distributed system’da juda muhim.

Chunki request:

• timeout bo‘lishi mumkin;

• client retry qilishi mumkin;

• Kafka message qayta kelishi mumkin;

• payment callback ikki marta kelishi mumkin.

6.1. Yomon misol

@PostMapping("/payments")
public PaymentResponse pay(@RequestBody PaymentRequest request) {
    paymentService.charge(request.userId(), request.amount());
    return new PaymentResponse("SUCCESS");
}

Agar client timeout oldi va yana yubordi:

1-request: userdan 100$ yechildi
2-request: yana 100$ yechildi

Bu juda xavfli.

6.2. To‘g‘ri yondashuv: Idempotency key

Client har payment uchun unique key yuboradi:

POST /payments
Idempotency-Key: pay_123456

Backend tekshiradi:

@PostMapping("/payments")
public PaymentResponse pay(
        @RequestHeader("Idempotency-Key") String key,
        @RequestBody PaymentRequest request
) {
    return paymentService.chargeOnce(key, request);
}

Service:

@Transactional
public PaymentResponse chargeOnce(String key, PaymentRequest request) {
    Optional<Payment> existing = paymentRepository.findByIdempotencyKey(key);

    if (existing.isPresent()) {
        return PaymentResponse.from(existing.get());
    }

    Payment payment = paymentGateway.charge(request.userId(), request.amount());

    paymentRepository.save(payment.withIdempotencyKey(key));

    return PaymentResponse.from(payment);
}

Bitta key bilan request 10 marta kelsa ham bitta payment bo‘ladi.

7. At-least-once delivery

Message brokerlarda, ayniqsa Kafka/RabbitMQ’da muhim tushuncha:

Message kamida bir marta yetkaziladi. Lekin ba’zida ikki marta ham kelishi mumkin.

Bu at-least-once delivery deyiladi.

Masalan:

Order Service → Kafka → Notification Service

Notification Service message oldi, email yubordi, lekin offset commit qilishdan oldin o‘chib qoldi.

Qayta ishga tushganda o‘sha message yana keladi.

Natija:

Userga 2 marta email ketishi mumkin

Shuning uchun consumer ham idempotent bo‘lishi kerak.

7.1. Kafka consumer’da idempotency

Yomon:

@KafkaListener(topics = "order-created")
public void handle(OrderCreatedEvent event) {
    emailService.sendOrderEmail(event.orderId());
}

Message qayta kelsa, email qayta ketadi.

Yaxshiroq:

@KafkaListener(topics = "order-created")
public void handle(OrderCreatedEvent event) {
    if (processedEventRepository.existsByEventId(event.eventId())) {
        return;
    }

    emailService.sendOrderEmail(event.orderId());

    processedEventRepository.save(new ProcessedEvent(event.eventId()));
}

Bu yerda eventId unique bo‘lishi kerak.

8. Distributed transaction muammosi

Monolithda transaction oson:

@Transactional
public void createOrder() {
    orderRepository.save(order);
    paymentRepository.save(payment);
    inventoryRepository.reserve(productId);
}

Bitta database bo‘lsa, hammasi bitta transaction ichida bo‘ladi.

Lekin microservice’da:

Order Service DB
Payment Service DB
Inventory Service DB

Endi bitta @Transactional hammasini qamrab olmaydi.

Masalan:

1. Order yaratildi
2. Payment muvaffaqiyatli bo‘ldi
3. Inventory service yiqildi

Endi nima bo‘ladi?

• orderni cancel qilasizmi?

• payment refund qilasizmi?

• retry qilasizmi?

• userga nima ko‘rsatasiz?

Mana shu distributed transaction muammosi.

9. 2PC — Two Phase Commit

2PC distributed transactionni qat’iy qilish usuli.

Unda coordinator bo‘ladi.

Phase 1: Prepare

Coordinator hamma servislardan so‘raydi:

Transactionga tayyormisan?

Servislar javob beradi:

Ha, tayyorman
Yo‘q, tayyor emasman

Phase 2: Commit/Rollback

Agar hamma tayyor bo‘lsa:

Commit

Agar bittasi tayyor bo‘lmasa:

Rollback

9.1. 2PC kamchiliklari

2PC kuchli consistency beradi, lekin production microservice’da ko‘p ishlatilmaydi.

Sabablari:

• sekin;

• coordinator single point of failure bo‘lishi mumkin;

• servislar block bo‘lib qoladi;

• network muammosida murakkab;

• scalability yomonlashadi.

Shuning uchun ko‘p real microservice tizimlarda 2PC o‘rniga SAGA ishlatiladi.

10. SAGA pattern

SAGA — distributed transactionni bir nechta lokal transactionlarga bo‘lish.

Har bir servis o‘z DB’sida lokal transaction qiladi. Agar keyingi qadamda xato bo‘lsa, oldingi qadam uchun compensating action bajariladi.

Masalan:

1. Order yaratish
2. Payment olish
3. Inventory reserve qilish
4. Delivery yaratish

Agar 3-qadamda xato bo‘lsa:

Payment refund
Order cancel

10.1. SAGA choreography

Choreography’da markaziy coordinator yo‘q. Servislar eventlar orqali gaplashadi.

Order Service: OrderCreated event
 ↓
Payment Service: PaymentCompleted event
 ↓
Inventory Service: InventoryReserved event
 ↓
Delivery Service: DeliveryCreated event

Agar xato bo‘lsa:

InventoryFailed event
 ↓
Payment Service refund qiladi
 ↓
Order Service cancel qiladi

Afzalligi:

• loosely coupled;

• event-driven;

• yaxshi scale bo‘ladi.

Kamchiligi:

• flow’ni tushunish qiyin;

• debugging murakkab;

• event chain chalkashib ketishi mumkin.

10.2. SAGA orchestration

Orchestration’da bitta coordinator bo‘ladi.

OrderSagaOrchestrator
  ↓
Payment Service
  ↓
Inventory Service
  ↓
Delivery Service

Coordinator qaysi qadam qachon bo‘lishini boshqaradi.

Afzalligi:

• flow aniq;

• debugging osonroq;

• biznes jarayon markazlashgan.

Kamchiligi:

• coordinator murakkablashadi;

• noto‘g‘ri yozilsa, monolit logicga aylanadi.

11. Outbox pattern

Distributed system’da juda muhim muammo bor:

@Transactional
public void createOrder(Order order) {
    orderRepository.save(order);
    kafkaTemplate.send("order-created", event);
}

Bu kod xavfli.

Nega?

Agar order DBga yozildi, lekin Kafka send bo‘lmadi — event yo‘qoladi.

Yoki Kafka send bo‘ldi, lekin DB rollback bo‘ldi — noto‘g‘ri event ketdi.

11.1. To‘g‘ri yechim: Outbox table

Bitta DB transaction ichida order va event saqlanadi:

@Transactional
public void createOrder(Order order) {
    orderRepository.save(order);

    outboxRepository.save(new OutboxEvent(
            UUID.randomUUID(),
            "OrderCreated",
            order.getId(),
            toJson(order)
    ));
}

Keyin alohida background publisher outbox’dan o‘qib Kafka’ga yuboradi:

outbox table → publisher → Kafka

Afzalligi:

• DB write va event write atomic bo‘ladi;

• event yo‘qolmaydi;

• retry qilish mumkin;

• distributed transaction kerak emas.

Senior Java backendda bu juda muhim pattern.

12. Circuit breaker

Circuit breaker — tashqi servis yiqilganda o‘z servisingizni ham yiqilishdan saqlaydigan pattern.

Masalan:

Order Service → Payment Service

Payment Service sekin yoki down bo‘lsa, Order Service ham threadlarini band qilib qo‘yadi.

Circuit breaker nima qiladi?

Agar xatolar ko‘paysa:
  circuit OPEN bo‘ladi
  requestlar Payment Servicega yuborilmaydi
  darhol fallback response qaytadi

12.1. Circuit breaker state’lari

12.2. Resilience4j bilan misol

@CircuitBreaker(name = "paymentService", fallbackMethod = "paymentFallback")
public PaymentResponse charge(PaymentRequest request) {
    return paymentClient.charge(request);
}

public PaymentResponse paymentFallback(PaymentRequest request, Throwable ex) {
    return new PaymentResponse("PAYMENT_TEMPORARILY_UNAVAILABLE");
}

Bu Payment Service yiqilganda Order Service’ni himoya qiladi.

13. Timeout

Distributed system’da timeout majburiy.

Yomon:

restTemplate.getForObject(url, PaymentResponse.class);

Agar timeout sozlanmagan bo‘lsa, request uzoq osilib qolishi mumkin.

To‘g‘ri:

@Bean
public RestClient restClient() {
    return RestClient.builder()
            .requestFactory(clientHttpRequestFactory())
            .build();
}

Timeoutlar:

connection timeout: ulanishga qancha kutamiz
read timeout: javobga qancha kutamiz

Qoida:

Timeout yo‘q servis — productionda bomba.

14. Retry

Retry vaqtinchalik xatolarda foydali:

• network glitch;

• 503;

• timeout;

• broker temporary unavailable.

Lekin retry xavfli bo‘lishi ham mumkin.

Yomon holat:

Payment Service sekin
1000 request retry qiladi
Payment Service yanada eziladi
Butun tizim sekinlashadi

Shuning uchun retry:

• limit bilan;

• backoff bilan;

• jitter bilan;

• faqat idempotent operationlarda;

• circuit breaker bilan birga ishlatiladi.

14.1. Retry + backoff

@Retry(name = "paymentRetry")
public PaymentResponse charge(PaymentRequest request) {
    return paymentClient.charge(request);
}

Config:

resilience4j:
  retry:
    instances:
      paymentRetry:
        max-attempts: 3
        wait-duration: 500ms
        enable-exponential-backoff: true

15. Backpressure

Backpressure — consumer yoki servis ortiqcha yukni ko‘tara olmasa, upstream’ga bosimni kamaytirishni bildirish.

Oddiyroq:

Men bunchalik tez qabul qila olmayman. Sekinroq yubor.

Masalan:

Producer sekundiga 10 000 message yuboryapti
Consumer sekundiga 2 000 message ishlayapti

Agar backpressure bo‘lmasa:

• queue to‘ladi;

• memory oshadi;

• latency oshadi;

• oxiri tizim yiqiladi.

15.1. Backpressure qayerda kerak?

• Kafka consumer;

• RabbitMQ consumer;

• WebFlux;

• batch processing;

• file import;

• notification sending;

• external API calls;

• database write.

Misol:

Flux.fromIterable(users)
        .flatMap(user -> sendNotification(user), 10)
        .subscribe();

Bu yerda 10 — parallel requestlar soni. Hammasini birdaniga yubormaydi.

16. Rate limiting

Rate limiting — client yoki servisga ma’lum vaqt ichida limit qo‘yish.

Masalan:

1 user → 1 minutda 100 request
1 IP → 1 minutda 500 request
1 API key → 1 sekundda 10 request

Nima uchun kerak?

• DDoSga qarshi;

• abuse oldini olish;

• expensive endpointlarni himoya qilish;

• fair usage;

• downstream servisni himoya qilish.

16.1. Rate limiting turlari

Fixed window

12:00:00 - 12:00:59 → 100 request
12:01:00 - 12:01:59 → 100 request

Oddiy, lekin oynalar chegarasida burst bo‘lishi mumkin.

Sliding window

Oxirgi 60 sekundga qaraydi.

Aniqroq, lekin murakkabroq.

Token bucket

Client token yig‘adi, request token sarflaydi.

Burstga ruxsat beradi, lekin umumiy limitni ushlaydi.

16.2. Spring’da oddiy rate limit konsept

Real productionda Redis asosida qilinadi.

key = rate:user:123
increment
expire 60 seconds
if count > limit reject

Response:

HTTP/1.1 429 Too Many Requests
Retry-After: 30

17. Distributed system’da observability majburiy

Distributed system’da bitta request bir nechta servisdan o‘tadi.

Gateway → Order → Payment → Inventory → Notification

Agar xato bo‘lsa, oddiy log yetmaydi.

Kerak bo‘ladi:

• correlation ID;

• structured logging;

• distributed tracing;

• metrics;

• alerting;

• dashboard.

17.1. Correlation ID

Har requestga bitta ID beriladi:

X-Correlation-Id: abc-123

Hamma servis logida shu ID bo‘ladi.

[abc-123] Order created
[abc-123] Payment started
[abc-123] Payment timeout

Shunda butun flow’ni topish oson.

18. Real hayotiy scenario

Tasavvur qiling ecommerce checkout bor:

POST /checkout

Flow:

1. Order yaratish
2. Payment qilish
3. Inventory reserve qilish
4. Notification yuborish

Yaxshi distributed design:

Client
 ↓
Order Service
 ↓
Order DB + Outbox
 ↓
Kafka: OrderCreated
 ↓
Payment Service
 ↓
Kafka: PaymentCompleted
 ↓
Inventory Service
 ↓
Kafka: InventoryReserved
 ↓
Notification Service

Muhim joylar:

• Order creation idempotent bo‘lishi kerak;

• Kafka consumer duplicate message’ga tayyor bo‘lishi kerak;

• Payment’da idempotency key bo‘lishi kerak;

• Inventory strong consistency talab qilishi mumkin;

• Notification eventual consistency bo‘lsa bo‘ladi;

• Outbox event yo‘qolishini kamaytiradi;

• retry/circuit breaker tashqi servislarni himoya qiladi;

• tracing butun flow’ni kuzatadi.

19. Senior developer bilishi kerak bo‘lgan savollar

Interview yoki real projectda shularni tushuntira olish kerak:

CAP

• CAP theorem nima?

• Consistency va availability qachon tanlanadi?

• Bank tizimida qaysi biri muhim?

• Notification service’da qaysi biri yetarli?

Idempotency

• Duplicate request kelsa nima bo‘ladi?

• Payment ikki marta yechilmasligi uchun nima qilasiz?

• Kafka message ikki marta kelsa consumer qanday ishlaydi?

SAGA

• 2PC va SAGA farqi nima?

• Choreography va orchestration farqi nima?

• Compensating transaction nima?

• Payment bo‘ldi, inventory failed — nima qilasiz?

Resilience

• Circuit breaker nima?

• Retry qachon xavfli?

• Timeout nima uchun majburiy?

• Rate limit qayerda qo‘yiladi?

• Backpressure bo‘lmasa nima bo‘ladi?

20. Qisqa xulosa

Distributed systems fundamentals Senior Java developer uchun majburiy skill.

Asosiy fikrlar:

Network is unreliable
Duplicate messages happen
Timeouts happen
Partial failure happens
Data may be temporarily inconsistent
Retries can make things worse
Observability is not optional

Senior developer distributed system’da quyidagilarni to‘g‘ri loyihalaydi:

• idempotency;

• retry;

• timeout;

• circuit breaker;

• SAGA;

• outbox pattern;

• consistency model;

• rate limiting;

• backpressure;

• tracing/logging.

Eng muhim qoida:

Distributed system’da “hammasi muvaffaqiyatli bo‘ladi” deb emas, “qaysi joyda yiqilishi mumkin va yiqilganda nima qilamiz?” deb design qilinadi.