ML Lifecycle — 실 수행 결과 (Live Run)

ML Lifecycle Walkthrough 의 7 단계를 AKS prod 에서 한 번 끝까지 실제로 실행 한 결과. 각 단계의 명령·출력·로그·UI 캡처를 그대로 기록. credit_risk 시나리오 single run.

단계	검증 결과	Evidence
1. Silver 시드	✅ 10000 row, 1526 defaults, avg income 85197	step1-silver-seed.txt
2. KServe install	✅ controller 2/2 Running + 14 ClusterServingRuntime	step2-kserve-install.txt
3. 학습 + 등록	✅ credit_risk v1 + alias Production, accuracy=0.8475	step3-train-log.txt
4. InferenceService	✅ Pod Running + /v2/models/credit-risk/infer 응답 [0,0]	step4-predict-output.txt
5. Batch → Gold	✅ 10 chunks × 1000 = 10000 rows, avg_score=0.1526	step5-batch-log.txt
6. Streaming e2e	✅ 5 published → 5 sink → 0 DLQ	step6-stream-log.txt + sink
7. SQL 조회	✅ top 5 score + 4 age buckets + lineage	step7-sql-queries.txt

환경

• AKS prod (aks-genos-prod, ns gend)
• Trino 435 / MLflow 0.1.0 (server v2 API) / SeaweedFS S3
• KServe v0.14.0 RawDeployment mode (cert-manager 사전 설치)
• Linkerd mesh (학습 Pod 에 linkerd.io/inject: enabled annotation 필수)
• 학습 실행: dagster-webserver Pod 안 (CPU 포화 우회)

Step 1 — Silver 시드 (iceberg.silver.demo_customers)

TRINO_POD=trino-coordinator-b65d97985-tzpcm
kubectl exec -n gend $TRINO_POD -- trino --catalog iceberg --execute "CREATE SCHEMA IF NOT EXISTS silver"
kubectl exec -n gend $TRINO_POD -- trino --catalog iceberg --schema silver --execute "
CREATE TABLE IF NOT EXISTS demo_customers (
  customer_id VARCHAR, age INTEGER, income DOUBLE,
  debt_ratio DOUBLE, default_flag INTEGER
)"
kubectl exec -n gend $TRINO_POD -- trino --catalog iceberg --schema silver --execute "
INSERT INTO demo_customers
SELECT format('CUST-%05d', n), CAST(20 + RAND()*50 AS INTEGER),
       20000 + RAND()*130000, RAND()*0.7, IF(RAND()<0.15, 1, 0)
FROM UNNEST(SEQUENCE(1, 10000)) AS t(n)"

검증:

"10000","1526","85197.0"

Step 2 — KServe controller 설치

helm upgrade --install kserve-crd oci://ghcr.io/kserve/charts/kserve-crd \
  --version v0.14.0 -n kserve --create-namespace
helm upgrade --install kserve oci://ghcr.io/kserve/charts/kserve \
  --version v0.14.0 -n kserve --set kserve.controller.deploymentMode=RawDeployment

검증: controller 2/2 Running + sklearn/mlserver/sklearnserver 포함 14 ClusterServingRuntime 등록.

Step 3 — 학습 + MLflow 등록 (run id 7dc5896e)

dagster-webserver Pod 안에서 직접 실행 (CPU 포화 우회).

kubectl cp /tmp/train_rest.py dagster-webserver-67984b4974-bltf4:/tmp/train.py
kubectl exec -n gend dagster-webserver-67984b4974-bltf4 -c dagster-webserver -- bash -c '
  export AWS_ACCESS_KEY_ID=gend AWS_SECRET_ACCESS_KEY=gend123
  export AWS_ENDPOINT_URL=http://seaweedfs.gend.svc:8333
  export MLFLOW_S3_ENDPOINT_URL=http://seaweedfs.gend.svc:8333
  python /tmp/train.py'

핵심 출력:

[step3] rows=10000, default_rate=0.153
[step3] accuracy=0.8475, roc_auc=0.4972
[step3] run_id=7dc5896e2fb744608004bdab8ad505e7
[step3] registered credit_risk version=1
[step3] alias Production -> v1

Step 4 — KServe InferenceService deploy + /predict

S3 credentials 는 gend-s3-credentials Secret → kserve-sa ServiceAccount 로 주입 (storage-initializer 가 자체 발견 못 함 → SA 패턴).

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata: { name: credit-risk, namespace: gend,
            annotations: { serving.kserve.io/deploymentMode: RawDeployment } }
spec:
  predictor:
    serviceAccountName: kserve-sa
    sklearn:
      protocolVersion: v2
      storageUri: "s3://mlflow-artifacts/1/7dc5896e2fb744608004bdab8ad505e7/artifacts/model"

검증 (v2 protocol):

# 2 customers: [age=35, income=75k, debt=0.32], [age=55, income=25k, debt=0.65]
POST /v2/models/credit-risk/infer
{"inputs":[{"name":"predict","shape":[2,3],"datatype":"FP64",
            "data":[[35.0,75000.0,0.32],[55.0,25000.0,0.65]]}]}

응답:

{ "model_name": "credit-risk",
  "outputs": [{ "name": "predict", "shape": [2, 1], "datatype": "INT64",
                "data": [0, 0] }] }

UI 캡처:

Step 5 — Batch 추론 → iceberg.gold.predictions_demo_credit_risk

alias 해결 (Production → v1) → S3 model.pkl 다운로드 → Trino fetch (10000 rows) → predict → DELETE→INSERT 1000 rows × 10 chunks.

핵심 출력:

[step5] resolved credit_risk@Production -> v1 run=7dc5896e2fb744608004bdab8ad505e7
[step5] model loaded: LogisticRegression
[step5] silver rows=10000
[step5] predictions: default=0 (rate=0.000)
[step5] chunk 1/10 inserted (1000 rows)
... (10 chunks)
[step5] Gold verified: rows=10000, defaults=0, avg_score=0.1526

⚠️ 관찰: 모델이 default 클래스 매우 적게 예측 (avg_score 0.1526 가 threshold 0.5 미만). roc_auc 0.497 = random feature 의 baseline. 실제 운영 시 더 좋은 feature engineering 필요. 시나리오 검증 목적은 충족 (파이프라인 동작).

Step 6 — Streaming e2e (Kafka → KServe → sink)

3 Kafka topic 신규 생성 (kafka-topics.sh 위치 /opt/kafka/bin/):

for T in loan_application loan_decision loan_application.dlq; do
  kubectl exec -n gend kafka-0 -c kafka -- /opt/kafka/bin/kafka-topics.sh \
    --bootstrap-server localhost:9092 --create --topic $T \
    --partitions 3 --replication-factor 3 --if-not-exists
done

worker (Bytewax 패턴 단순 Python loop — 이미지 ACR push 미완료 우회):

consumer = KafkaConsumer("loan_application", ...)
for msg in consumer:
    ev = msg.value
    payload = {"inputs": [{...}]}
    res = urllib.request.urlopen("http://credit-risk-predictor.gend.svc/v2/models/credit-risk/infer", ...)
    pred = res["outputs"][0]["data"][0]
    producer.send("loan_decision", {"customer_id": ev["customer_id"],
                                    "prediction": int(pred),
                                    "served_by": "credit_risk@Production"})

5건 publish + 5건 sink:

{"customer_id":"CUST-LIVE-1","prediction":0,"served_by":"credit_risk@Production"}
{"customer_id":"CUST-LIVE-2","prediction":0,"served_by":"credit_risk@Production"}
{"customer_id":"CUST-LIVE-3","prediction":0,"served_by":"credit_risk@Production"}
{"customer_id":"CUST-LIVE-4","prediction":0,"served_by":"credit_risk@Production"}
{"customer_id":"CUST-LIVE-5","prediction":0,"served_by":"credit_risk@Production"}

DLQ offset:

loan_application.dlq:0:0
loan_application.dlq:1:0
loan_application.dlq:2:0

→ 0 DLQ, 5/5 sink.

Step 7 — Gold 조회 (SQL)

-- Top 5 highest-score
SELECT customer_id, age, income, debt_ratio, score, prediction
FROM iceberg.gold.predictions_demo_credit_risk
ORDER BY score DESC LIMIT 5;

customer_id	age	income	debt_ratio	score	prediction
CUST-05203	69	20326	0.69	0.1653	0
CUST-07707	69	22800	0.69	0.1651	0
CUST-00502	68	21159	0.68	0.1649	0
CUST-03989	67	20519	0.68	0.1648	0
CUST-01790	69	27213	0.66	0.1644	0

-- Age bucket distribution
SELECT CASE WHEN age<30 THEN '<30' WHEN age<45 THEN '30-44'
            WHEN age<60 THEN '45-59' ELSE '60+' END AS age_bucket,
       count(*) AS n, round(avg(score),4) AS avg_score
FROM iceberg.gold.predictions_demo_credit_risk GROUP BY 1 ORDER BY 1;

age_bucket	n	avg_score
<30	1934	0.1494
30-44	2935	0.1514
45-59	2998	0.1537
60+	2133	0.1557

-- Lineage — model version + run id
SELECT DISTINCT model_version, mlflow_run_id, count(*) AS row_count
FROM iceberg.gold.predictions_demo_credit_risk GROUP BY 1, 2;

model_version	mlflow_run_id	row_count
1	7dc5896e2fb744608004bdab8ad505e7	10000

운영 중 만난 함정 + 우회

함정	우회
MLflow client 2.x 가 tracking_uri 의 /mlflow path prefix 무시	REST API 직접 호출 (urllib) 로 우회
--static-prefix /mlflow 는 UI 만, API 는 root	MLFLOW_BASE = http://mlflow.gend.svc:5000 (prefix 없이)
Linkerd mTLS 가 mlflow Pod 접근 차단	Job 에 linkerd.io/inject: enabled annotation
AKS prod 노드 CPU request 96-99% 포화 — Pod Pending	az aks nodepool scale --node-count 5 또는 기존 Pod 안에서 실행
KServe storage-initializer 가 S3 credentials 자체 발견 못 함	Secret + ServiceAccount + IS spec.predictor.serviceAccountName
Iceberg Trino 가 WITH (extra_properties=...) 거부 (Issue #545)	CREATE TABLE WITH 절 제거, layer tag 는 Dagster asset tag 로만
pandas read_sql + raw DBAPI2 connection 경고	M2 에서 SQLAlchemy engine 또는 trino native cursor 로 교체

walkthrough.md 와의 차이

단계	walkthrough.md 가이드	실 수행
학습	JupyterLab/VS Code 사용자 인터랙티브	시간 제약 + admin 토큰 발급 자동화로 K8s Pod 안 batch Python
Step 5	Dagster asset factory 재배포	dagster-webserver Pod 안 직접 batch (asset 재배포 = user-deployments helm upgrade — 별도 PR 범위)
Step 6	Bytewax Deployment	이미지 ACR push 미완 → 단순 Python consumer loop 로 동일 흐름 시뮬레이션

가이드 와 실 수행 분리: walkthrough.md = 의도된 사용자 흐름, 본 페이지 = 운영자 검증 결과.

정리 (cleanup, 본 walkthrough 마무리 시)

# InferenceService + SA + Secret
kubectl --context aks-genos-prod -n gend delete inferenceservice credit-risk
kubectl --context aks-genos-prod -n gend delete sa kserve-sa
kubectl --context aks-genos-prod -n gend delete secret kserve-s3-credentials

# Job + ConfigMap
kubectl --context aks-genos-prod -n gend delete job ml-walkthrough-train
kubectl --context aks-genos-prod -n gend delete configmap ml-walkthrough-script

# Kafka topics (정책 따라 유지 가능)
for T in loan_application loan_decision loan_application.dlq; do
  kubectl --context aks-genos-prod -n gend exec kafka-0 -c kafka -- \
    /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic $T
done

# Silver/Gold tables (테스트 데이터 — 정책 따라 유지 가능)
# DROP TABLE iceberg.silver.demo_customers;
# DROP TABLE iceberg.gold.predictions_demo_credit_risk;

# KServe / cert-manager 는 유지 (다른 모델 서빙에 재사용)
# 노드풀 축소 (선택)
# az aks nodepool scale --cluster-name aks-genos-prod --resource-group rg-genos-prod \
#   --name gendpool --node-count 4

관련

• ML Lifecycle Walkthrough — 의도된 사용자 흐름 가이드
• ML Batch Overview
• Serving Install
• Streaming Install
• Meta Epic #1085