systemd

주요 명령어

systemd를 위한 주요 명령어는 다음과 같습니다.

  • systemctl
  • systemctl-analyze
  • systemctl-cgls
  • systemd-cgtop
  • systemd-loginctl

systemctl

전체 서비스 목록을 출력해 줍니다.

$ systemctl
...
  ipmi.service                                                                                      loaded active exited    IPMI Driver
  ipmievd.service                                                                                   loaded failed failed    Ipmievd Daemon
  irqbalance.service                                                                                loaded active running   irqbalance daemon
  kmod-static-nodes.service                                                                         loaded active exited    Create list of required static device nodes for the current kernel
  kubelet.service                                                                                   loaded active running   kubelet: The Kubernetes Node Agent
  lvm2-lvmetad.service                                                                              loaded active running   LVM2 metadata daemon
  lvm2-monitor.service                                                                              loaded active exited    Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling
  network.service                                                                                   loaded failed failed    LSB: Bring up/down networking
  noms_nsight.service                                                                               loaded active exited    SYSV: noms_nsight
  noms_nsight_jp.service                                                                            loaded active exited    SYSV: noms_nsight
  nscd.service                                                                                      loaded active running   Name Service Cache Daemon
  ntpdate.service                                                                                   loaded active exited    Set time via NTP
  prov2-line-agent.service                                                                          loaded active running   PROV2 agent
  rhel-dmesg.service                                                                                loaded active exited    Dump dmesg to /var/log/dmesg
  rhel-domainname.service                                                                           loaded active exited    Read and set NIS domainname from /etc/sysconfig/network
  rhel-import-state.service                                                                         loaded active exited    Import network configuration from initramfs
  rhel-readonly.service                                                                             loaded active exited    Configure read-only root support
  ...

변경된 서비스 설정 정보를 데몬에 반영하기

$ systemctl daemon-reload

서비스를 활성화 하기

$ systemctl enable [서비스명]

서비스를 비활성화 하기

$ systemctl disable [서비스명]

서비스 활성화 여부 확인하기

$ systemctl is-enabled [서비스명]
enabled

서비스 시작하기

 systemctl start [서비스명]

서비스 종료하기

$ systemctl stop [서비스명]

서비스 재시작하기

$ systemctl restart [서비스명]

서비스 갱신하기

$ systemctl reload [서비스명]

서비스 동작 여부 확인하기

$ systemctl is-active [서비스명]
active

서비스 상태 출력 하기

$ systemctl status -l [서비스명]
kubelet.service - kubelet: The Kubernetes Node Agent
 Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/kubelet.service.d
         └─10-kubeadm.conf
 Active: active (running) since Wed 2018-12-26 20:14:53 JST; 5 months 21 days ago
   Docs: https://kubernetes.io/docs/
 Main PID: 77429 (kubelet)
  Tasks: 24
 Memory: 68.2M
 CGroup: /system.slice/kubelet.service
         └─77429 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni

서비스 이름과 활성화 여부 확인하기

$ systemctl list-unit-files
...
initrd-udevadm-cleanup-db.service             static
ip6tables.service                             disabled
ipmi.service                                  enabled
ipmievd.service                               enabled
iprdump.service                               disabled
iprinit.service                               disabled
iprupdate.service                             disabled
ipsec.service                                 disabled
iptables.service                              enabled
irqbalance.service                            enabled
kdump.service                                 masked
kmod-static-nodes.service                     static
kpatch.service                                disabled
kubelet.service                               enabled
libstoragemgmt.service                        masked
lm_sensors.service                            masked
...

부팅시 실패한 서비스 목록 출력하기

$ systemctl --failed
  UNIT                           LOAD   ACTIVE SUB    DESCRIPTION
  ipmievd.service                loaded failed failed Ipmievd Daemon
  network.service                loaded failed failed LSB: Bring up/down networking
  systemd-tmpfiles-clean.service loaded failed failed Cleanup of Temporary Directories
  systemd-tmpfiles-setup.service loaded failed failed Create Volatile Files and Directories

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

4 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

systemd-analyze

systemd-analyze 명령어는 분석 정보를 출력해 줍니다.

$ systemd-analyze
Startup finished in 862ms (kernel) + 868ms (initrd) + 1min 4.908s (userspace) = 1min 6.639s
$ systemd-analyze blame
         50.457s unbound-anchor.service
         30.003s iaas-pm-bootup.service
          7.705s cloud-final.service
          7.102s cloud-init.service
          3.270s ntpdate.service
          2.743s network.service
          1.252s docker.service
           640ms cloud-init-local.service
           369ms cloud-config.service
           345ms lvm2-monitor.service
           313ms systemd-udev-settle.service
           251ms dev-vda2.device
           183ms prov2-line-agent.service
           141ms snmpd.service
           124ms sshd-keygen.service
           111ms hostagent_kr.service
           111ms hostagent_jp_line.service
            90ms systemd-vconsole-setup.service
            51ms rc-local.service
...

critical-chain을 이용하면, 시간을 많이 소요한 서비스의 실행과 대기에 대해서 체인 형태로 출력할 수 있습니다.

$ systemd-analyze critical-chain
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

multi-user.target @34.062s
└─iaas-pm-bootup.service @4.056s +30.003s
  └─basic.target @623ms
    └─sockets.target @623ms
      └─dbus.socket @623ms
        └─sysinit.target @622ms
          └─systemd-update-utmp.service @617ms +5ms
            └─systemd-remount-fs.service @183ms +13ms
              └─systemd-fsck-root.service @584542y 2w 2d 20h 1min 49.069s +11ms
                └─systemd-journald.socket
                  └─-.slice

systemd-cgls

systemd-cgls 명령어는 현재 cgroup에 대한 정보를 타입별로 출력해줍니다.

$ systemd-cgls
├─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
├─kubepods
│ └─burstable
│   ├─podfc4890ab91dfb8b8a630043af37f52d8
│   │ ├─dc2312d450abd72c1490775e615fbf41abbef49bb4ccdc6a9d9d8bb7ec880724
│   │ │ └─40435 kube-apiserver --apiserver-count=5 --authorization-mode=Node,RBAC --advertise-address=10.xx.xx.xx --allow-privileged=true --client-ca-file=/etc/kubernetes/pki/ca.crt --disable-admission-plugins=Persiste
│   │ └─5b42977c143110757f6d7ae747fbe435f878f8427d30274936eb06bb5dc21b2c
│   │   └─40405 /pause
│   ├─pod734afb7f-08db-11e9-aa26-fa1648a9a65b
│   │ ├─a76db3d9d75d1a48f3231c75b601fd6c7ad1b33ff83e054482b800b2cf6ca049
│   │ │ └─104263 /bin/node_exporter --path.procfs=/host/proc --path.sysfs=/host/sys
│   │ └─e8bffecd18687c7cb1ca90b3de906928e6af50081601203a2bab7f5fea19ae34
│   │   └─104169 /pause
│   ├─pod49e68627-08ce-11e9-837e-fa1648a9a65b
│   │ ├─dd6d53bb4036a64572e4a64f5997a87e5bfb6e5c25f453153854fca88105e54a
│   │ │ └─kube-proxy
│   │ │   └─80946 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf
│   │ └─bc202979fd26611a1ec0ac59fd013d66dff2262da590d326dd3817ed5a56dd28
│   │   └─80907 /pause
│   ├─pod45de6e05-08ce-11e9-837e-fa1648a9a65b
│   │ ├─3881a2163345cf5b6f9c6bca07e2f36394fe6a0ee3a13a33a3f12e9669385a52
│   │ │ └─80804 /opt/bin/flanneld --ip-masq --kube-subnet-mgr --iface=private --iface=bond0 --iface=eth0
│   │ └─a8b41a45c4e3cf278d3841e67914c81382bc4b536f6b86183c3a2eed4f059377
│   │   └─80713 /pause
│   ├─pod705e7ce1217a37349a5567101e60165d
│   │ ├─51d6579bb1d26c0868908077434fe519377b67ba5bd81530bd3d0c3ad1369260
│   │ │ └─82155 kube-scheduler --address=127.0.0.1 --kubeconfig=/etc/kubernetes/scheduler.conf --leader-elect=true
│   │ └─dfc5ef7ff3565013fd50361c2a848010916fe4246dbd6c190df740068648c510
│   │   └─77628 /pause
...

systemd-cgtop

system-cgtop 명령어는 흔히 알고 있는 top 명령어 처럼, cgroup에 대하여 CPU, Memory, I/O에 대한 결과를 출력해 줍니다.

$ $ systemd-cgtop
Path                                                                                         Tasks   %CPU   Memory  Input/s Output/s

/                                                                                               84      -     4.8G        -        -
/kubepods.slice                                                                                  -      -   850.1M        -        -
/kubepods.slice/kubepods-besteffort.slice                                                        -      -   370.7M        -        -
/kubepods.slice/kubepods-best...s-besteffort-pod0b4c1d31_78dc_4d84_b496_6f082c63977b.slice       -      -    14.1M        -        -
/kubepods.slice/kubepods-best...epods-besteffort-pod0eb79a133166edcb3192c85c01bd9b32.slice       -      -   356.6M        -        -
/kubepods.slice/kubepods-burstable.slice                                                         -      -   463.2M        -        -
/kubepods.slice/kubepods-burs...bepods-burstable-pod4c9fe2c16888e009cff100467a01a432.slice       -      -    14.8M        -        -
/kubepods.slice/kubepods-burs...bepods-burstable-pod87880e44c58a3215aba1a2b52b5f211e.slice       -      -    14.1M        -        -
/kubepods.slice/kubepods-burs...bepods-burstable-podbe31476e2cd67aa8bdc87bbf4f55679d.slice       -      -   434.2M        -        -
/kubepods.slice/kubepods-podb3a9dde5_43ee_43db_af71_e0d4cfb0e27c.slice                           -      -    16.1M        -        -
/kubepods.slice/kubepods-podb...9d86c2566ee644a239924a32c90888bb76baa2ab1986ac3e54e7.scope       1      -    16.0M        -        -
/kubepods.slice/kubepods-podb...5a75cafc6ee2540f1110c5aabfc177d03df2291c4229949adc96.scope       1      -    44.0K        -        -
/system.slice                                                                                    -      -     1.3G        -        -

디스크 사용량

df : 디스크 사용량 확인

디스크의 사용량을 보기 좋게 출력해 준다

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda2       100G  6.6G   94G   7% /
devtmpfs         16G     0   16G   0% /dev
tmpfs            16G   12K   16G   1% /dev/shm
tmpfs            16G 1004K   16G   1% /run
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/vdc1       3.0T  528G  2.5T  18% /mnt
tmpfs           3.2G     0  3.2G   0% /run/user/0
tmpfs           3.2G     0  3.2G   0% /run/user/10000
...

현재 디렉토리를 포함하는 파티션의 용량을 출력해 준다.

$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdc1       3.0T  528G  2.5T  18% /mnt

du: 디렉토리 및 파일 용량 확인

현재 디렉토리에 있는, 디렉토리 및 파일 용량을 보기 좋게 출력해 준다.

$ du -sh *
41G     01DAR7DQAV4TX9XMQA53H8P2KN
38G     01DCJ573XV9BS33MW7728Y4YR1
38G     01DCQYKT2QCZHF9J00H2W2GV62
38G     01DCXR075DH20RB24RAY92M129
1.6G    01DCZN81WMAWP6RCD79XKV8T0J
13G     01DCZNJW2WSNXC6GYNFY1AEH9M
1.7G    01DCZW3S4K9BM13KS623FDTZ26
0       lock
31G     wal

현재 디렉토리에 있는, 디렉토리 및 파일 중에서 용량이 큰 순서대로 10개를 출력해 준다.

$ du -sh * | sort -rh | head -n 10

etcd 명령어 예제

인증서 기반

TLS 인증서 기반으로 etcd를 설치한 경우, etcdctl을 사용하려면, 인증서 정보를 플래그로 넘겨줘야 합니다. 그리고 버전 3의 API를 사용하려면 ETCDCTL_API=3을 선언해줘야 합니다.

ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
    --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
    get foo

명령어가 상당히 길기 때문에, alias로 지정해 놓고 사용하면 편리합니다.

alias etcd3ctl="ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key"

명령어 예제

전체 조회

버전 3 부터는 ls 명령어가 존재하지 않습니다. 전체 목록 같은 것을 조회하려면, --prefix 플래그를 사용할 수 있습니다.

아래와 같이 실행하면, 전체 목록을 조회해 볼 수 있습니다.

$ etcd3ctl get / --prefix --keys-only

...
/registry/deployments/kube-system/coredns
/registry/deployments/kube-system/metrics-server
/registry/deployments/kube-system/tiller-deploy
/registry/deployments/openebs/maya-apiserver
/registry/deployments/openebs/openebs-admission-server
/registry/deployments/openebs/openebs-localpv-provisioner
/registry/deployments/openebs/openebs-provisioner
/registry/deployments/openebs/openebs-snapshot-operator
/registry/deployments/weave/weave-scope-app
...

멤버 조회

$ etcd3ctl member list
67fab7a197a31464, started, etcd-001, https://10.x.x.x:2380, https://10.x.x.x:2379
9ebdbf1241485ebd, started, etcd-002, https://10.y.y.y:2380, https://10.y.y.y:2379
f3aacf2611e12d71, started, etcd-003, https://10.z.z.z:2380, https://10.z.z.z:2379

엔드 포인트 상태

$ etcd3ctl endpoint health
https://[127.0.0.1]:2379 is healthy: successfully committed proposal: took = 1.13549ms

$ etcd3ctl endpoint status
https://[127.0.0.1]:2379, f3aacf2611e12d71, 3.3.10, 4.6 MB, false, 6, 6880100

etcd 모니터링 하기

Metrics Endpoint

각각의 etcd 서버는 클라이언트 접속용 포트의 /metrics 경로를 이용해서 메트릭을 출력해 줍니다. (클라이언트용 기본 포트는 2379 입니다.)

$ curl http://127.0.0.1:2379/metrics

...
tcd_disk_wal_fsync_duration_seconds_count 4.9931728e+07
# HELP etcd_grpc_proxy_cache_hits_total Total number of cache hits
# TYPE etcd_grpc_proxy_cache_hits_total gauge
etcd_grpc_proxy_cache_hits_total 0
# HELP etcd_grpc_proxy_cache_keys_total Total number of keys/ranges cached
# TYPE etcd_grpc_proxy_cache_keys_total gauge
etcd_grpc_proxy_cache_keys_total 0
# HELP etcd_grpc_proxy_cache_misses_total Total number of cache misses
# TYPE etcd_grpc_proxy_cache_misses_total gauge
etcd_grpc_proxy_cache_misses_total 0
# HELP etcd_grpc_proxy_events_coalescing_total Total number of events coalescing
# TYPE etcd_grpc_proxy_events_coalescing_total counter
etcd_grpc_proxy_events_coalescing_total 0
# HELP etcd_grpc_proxy_watchers_coalescing_total Total number of current watchers coalescing
# TYPE etcd_grpc_proxy_watchers_coalescing_total gauge
etcd_grpc_proxy_watchers_coalescing_total 0
# HELP etcd_mvcc_db_total_size_in_bytes Total size of the underlying database physically allocated in bytes.
# TYPE etcd_mvcc_db_total_size_in_bytes gauge
etcd_mvcc_db_total_size_in_bytes 1.0145792e+07
# HELP etcd_mvcc_db_total_size_in_use_in_bytes Total size of the underlying database logically in use in bytes.
# TYPE etcd_mvcc_db_total_size_in_use_in_bytes gauge
etcd_mvcc_db_total_size_in_use_in_bytes 4.77184e+06
...

하지만 TLS 인증 기반으로 etcd를 설치했을 경우, 단순히 매트릭을 보기 위해서도 인증서가 필요합니다. 이런 불편한 때문에 etcd v3.3.0 부터는 별도의 /metrics 엔드포인트를 지정할 수 있는 기능이 생겼습니다. 엔드포인트를 지정할 수 있는 가장 쉬운 방법은 바로 --listen-metrics-urls 플래그 입니다. 또 다른 방법으로는 환경 변수나, 설정 파일에 ETCD_LISTEN_METRICS_URLS 값을 설정하는 것입니다.

예를 들어 --listen-metrics-urls=http://0.0.0.0:2381 플래그를 etcd를 실행 할 때 추가하면, 2381 포트로 /metrics/health 를 조회해 볼 수 있습니다.

Health Endpoint

/health 엔드포인트를 이용하면, etcd 서버의 상태를 알 수 있습니다.

$ curl http://127.0.0.1:2379/health
{"health":"true"}

Prometheus

프로메테우스 설정 파일에, etcd 스크랩 설정을 추가하면, 프로메테우스가 etcd의 메트릭 엔드포인트에 접근하여, 메트릭 데이터를 수집해 갑니다.

scrape_configs:
  - job_name: etcd
    static_configs:
    - targets: ['x.x.x.x:2381','y.y.y.y:2381','z.z.z.z:2381']

참고 문서