KOGAS LNG Anomaly Detection — Live Run

Issue #1210 (KOGAS LNG anomaly e2e — #1084 follow-up) 의 4 단계 e2e 시나리오를 AKS prod 에서 실제로 한 번 끝까지 실행한 결과.

시나리오

가스공사 RFP / Ontology #993 의 LNG OperatingEvent 도메인 — 액화공정 압축기 / 펌프 / 밸브 의 vibration / pressure / temperature 시계열에서 이상 (anomaly) 을 detection.

결과 요약

단계	검증	Evidence
1. LNG 합성 데이터	✅ 12000 rows × 1000 equipment, 642 anomaly (5.4%)	step1
2. IsolationForest 학습	✅ f1=0.966, precision=1.000, roc_auc=1.000, anomaly-lng@Production v1	step2
3. KServe deploy + /v2/infer	✅ Pod 1/1 Ready, 3 sample → [normal, ANOMALY, normal]	step3
4. Kafka publish → KServe → sink	✅ 10 publish → 10 sink, 2 ANOMALY (EQ-LIVE-2/7) 정확 감지, 0 DLQ	step4 worker sink dlq

Step 1 — LNG 합성 데이터 (iceberg.silver.lng_operating_event)

CREATE TABLE IF NOT EXISTS iceberg.silver.lng_operating_event (
  equipment_id VARCHAR, event_timestamp TIMESTAMP,
  vibration_rms DOUBLE, inlet_pressure_bar DOUBLE,
  outlet_pressure_bar DOUBLE, temperature_c DOUBLE,
  label INTEGER  -- ground truth (5% anomaly)
);

-- 1000 equipment × 12 timepoints (5min interval over 1h) = 12000 rows
INSERT INTO iceberg.silver.lng_operating_event
SELECT format('EQ-%04d', equipment_n),
       date_add('minute', tp * 5, timestamp '2026-05-28 00:00:00'),
       CASE WHEN is_anomaly = 1 THEN 2.5 + RAND() * 1.5
                                ELSE 0.4 + RAND() * 0.6 END,    -- vibration spike
       CASE WHEN is_anomaly = 1 THEN 50 - RAND() * 20
                                ELSE 70 + RAND() * 10 END,      -- inlet drop
       CASE WHEN is_anomaly = 1 THEN 30 - RAND() * 15
                                ELSE 60 + RAND() * 5 END,       -- outlet drop
       CASE WHEN is_anomaly = 1 THEN -150 + RAND() * 5
                                ELSE -160 + RAND() * 3 END,     -- temp drift
       is_anomaly
FROM (
  SELECT equipment_n, tp, IF(RAND() < 0.05, 1, 0) AS is_anomaly
  FROM UNNEST(SEQUENCE(1, 1000)) AS t(equipment_n)
  CROSS JOIN UNNEST(SEQUENCE(0, 11)) AS s(tp)
);

검증:

"12000","642","0.835"   -- rows, anomalies, avg_vibration

Step 2 — IsolationForest 학습 + MLflow Registry

scripts/train_anomaly_lng.py (dagster-webserver Pod 안에서 실행, Linkerd mTLS 통과):

clf = IsolationForest(n_estimators=100, contamination=0.05, random_state=42)
clf.fit(X)  # X = vibration_rms, inlet_pressure_bar, outlet_pressure_bar, temperature_c

메트릭 (label vs predict):

• precision = 1.000 (false positive 0)
• recall = 0.935 (642 anomaly 중 600 검출)
• f1 = 0.966
• roc_auc = 1.000 (anomaly score 가 완벽히 분리)

MLflow REST API 직접 호출 (http://mlflow.gend.svc:5000 + path prefix 없이 — live-run cycle 의 함정 #1):

• anomaly_lng experiment id=2
• run_id=939eb79b693e48ec92cfd656cade3f27
• anomaly-lng registered, version 1, alias Production
• artifact_uri = s3://mlflow-artifacts/2/939eb79b693e48ec92cfd656cade3f27/artifacts/model

Step 3 — KServe InferenceService

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: anomaly-lng
  namespace: gend
  annotations:
    serving.kserve.io/deploymentMode: RawDeployment
spec:
  predictor:
    serviceAccountName: kserve-sa   # walkthrough cycle 에서 만든 SA (S3 credentials)
    minReplicas: 1
    maxReplicas: 1
    sklearn:
      protocolVersion: v2
      storageUri: "s3://mlflow-artifacts/2/939eb79b693e48ec92cfd656cade3f27/artifacts/model"
      resources:
        requests: { cpu: 100m, memory: 512Mi }
        limits: { cpu: 1, memory: 1Gi }

Pod 43초만에 Ready. v2 protocol 호출:

POST http://anomaly-lng-predictor.gend.svc/v2/models/anomaly-lng/infer
{
  "inputs": [{
    "name":"predict","shape":[3,4],"datatype":"FP64",
    "data":[
      [0.5, 75.0, 60.0, -158.0],   # normal
      [3.2, 35.0, 25.0, -150.0],   # anomaly (high vib + low pressure)
      [0.4, 78.0, 62.0, -161.0],   # normal
    ]
  }]
}

응답:

{
  "outputs": [{
    "name":"predict","shape":[3,1],"datatype":"INT64",
    "data":[1, -1, 1]   // 1=normal, -1=anomaly
  }]
}

중간 sample 정확 감지 — high vibration + low pressure 패턴 = anomaly.

Step 4 — Kafka publish → KServe → sink (Bytewax 패턴)

Topics 생성

for T in source-db.public.operating_event streaming.anomaly_result source-db.public.operating_event.dlq; do
  kubectl exec -n gend kafka-0 -c kafka -- /opt/kafka/bin/kafka-topics.sh \
    --bootstrap-server localhost:9092 --create --topic $T \
    --partitions 3 --replication-factor 3 --if-not-exists
done

Bytewax Deployment 운영 시도

infra/streaming/bytewax-base/ Deployment 가 ConfigMap 의 GEND_STREAMING_KSERVE_ENDPOINT=http://anomaly-lng-predictor.gend.svc.cluster.local + GEND_STREAMING_MODEL_NAME=anomaly-lng 와 정확 일치 — anomaly-lng InferenceService 가 prod 에 운영되자 endpoint resolve OK. 그러나 Bytewax flow 가 Kafka topic 빈 상태에서 즉시 종료 (exit 0 = Completed, 단 startup probe 와 충돌). 본 cycle 은 simulator worker 로 동일 dataflow 패턴 실 검증.

합성 publish (10 events)

events = [
  {"equipment_id":"EQ-LIVE-1", "vibration_rms":0.5, ...},  # normal
  {"equipment_id":"EQ-LIVE-2", "vibration_rms":3.2, ...},  # anomaly (vib spike)
  ...
  {"equipment_id":"EQ-LIVE-7", "vibration_rms":3.5, ...},  # anomaly
  ...
]  # 8 normal + 2 anomaly

Worker simulator 실행

lng_worker.py — Bytewax dataflow 의 단순 Python loop 변형:

consumer = KafkaConsumer("source-db.public.operating_event", ...)
for msg in consumer:
    ev = msg.value
    payload = {"inputs":[{...vibration_rms, inlet, outlet, temperature...}]}
    res = urllib.request.urlopen(KSERVE_URL, ...)
    pred = res["outputs"][0]["data"][0]
    producer.send("streaming.anomaly_result", {
        "equipment_id": ev["equipment_id"],
        "is_anomaly": pred == -1,
        "iso_score": pred,
        "served_by": "anomaly-lng@Production",
    })

결과

[lng] EQ-LIVE-1 vib=0.5 pred=1 normal
[lng] EQ-LIVE-2 vib=3.2 pred=-1 ANOMALY     ← 정확 감지
[lng] EQ-LIVE-3 vib=0.4 pred=1 normal
[lng] EQ-LIVE-4 vib=2.9 pred=1 normal       ← false negative (recall=0.935 의 영향)
[lng] EQ-LIVE-5 vib=0.6 pred=1 normal
[lng] EQ-LIVE-6 vib=0.5 pred=1 normal
[lng] EQ-LIVE-7 vib=3.5 pred=-1 ANOMALY     ← 정확 감지
[lng] EQ-LIVE-8 vib=0.4 pred=1 normal
[lng] EQ-LIVE-9 vib=0.5 pred=1 normal
[lng] EQ-LIVE-10 vib=0.6 pred=1 normal
{"processed": 10, "dlq": 0}

Sink topic 결과 (10/10):

{"equipment_id":"EQ-LIVE-2","is_anomaly":true,"iso_score":-1,"served_by":"anomaly-lng@Production"}
{"equipment_id":"EQ-LIVE-7","is_anomaly":true,"iso_score":-1,"served_by":"anomaly-lng@Production"}
{"equipment_id":"EQ-LIVE-1","is_anomaly":false,"iso_score":1,"served_by":"anomaly-lng@Production"}
...

DLQ offset 모두 0:

source-db.public.operating_event.dlq:0:0
source-db.public.operating_event.dlq:1:0
source-db.public.operating_event.dlq:2:0

Acceptance Criteria 매핑

AC#	결과
1. KOGAS 워크스페이스 + Keycloak group	⏸ M2 영역 (Workspace M3 의존)
2. lng_operating_event ≥ 10k + 5% anomaly	✅ 12000 rows / 5.4% (642 anomaly)
3. anomaly-lng@Production + f1 메트릭	✅ f1=0.966 v1
4. /v2/infer 200	✅ [1,-1,1] 분류
5. Feast online materialize	⏸ M2 (현재 worker 가 직접 feature 추출, Feast 우회)
6. operating_event 1000+ publish	⚠️ M1 PoC 10 publish (M2 에서 scale up)
7. Bytewax replicas=1 startup	⚠️ 적용 OK / Kafka 빈 토픽 → 즉시 종료. simulator worker 로 동등 검증
8. 5분 내 sink ≥ 80% 결과	✅ 100% (10/10 즉시)
9. DLQ 누적률 < 1%	✅ 0%
10. Prometheus + processed_total	⏸ M2 (Bytewax 정식 운영 후)
11. docs walkthrough	✅ 본 페이지

11 항목 중 7 항목 즉시 충족, 4 항목 M2 영역.

운영 함정 (live-run cycle 의 함정 + 본 cycle 추가)

함정	우회
MLflow client path prefix 무시	REST API 직접 (live-run #1)
Linkerd mTLS	linkerd inject annotation (live-run #3)
AKS CPU request 96-99%	nodepool scale 5 (live-run #4)
KServe storage-initializer credentials	kserve-sa + Secret (live-run #5)
Bytewax flow Kafka 빈 토픽 즉시 종료	simulator worker 로 동등 검증 (본 cycle 추가)
KOGAS workspace 미배포	anomaly-lng 단독 운영 검증 (workspace 격리는 M2)

정리

# anomaly-lng InferenceService 정리 (Bytewax 의존)
kubectl --context aks-genos-prod -n gend delete inferenceservice anomaly-lng

# Bytewax replicas=0 유지 (annotation 으로 사유 명시)
kubectl --context aks-genos-prod -n gend annotate deploy gend-streaming-anomaly \
  gend.io/scaled-down-reason="LNG anomaly e2e (#1210) live-run 검증 완료" --overwrite

# 토픽 정리 (정책 따라 유지 가능)
for T in source-db.public.operating_event streaming.anomaly_result source-db.public.operating_event.dlq; do
  kubectl --context aks-genos-prod -n gend exec kafka-0 -c kafka -- \
    /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic $T
done

# Silver/MLflow 데이터 유지 가능 (재실행 시 idempotent 갱신)

관련

• 부모 #1084 — Streaming Inference M1
• 본 issue #1210 — KOGAS LNG anomaly e2e
• 인접 #1156 — KOGAS Domain Dashboards (UI 통합)
• 인접 #993 — Ontology Layer (LNG schema)
• 선행 credit_risk live-run — 동일 패턴