logback + fluent-bit + opensearch 개발 서버용

로그를 편하게 한곳에서 볼 수 있도록 로그 수집기를 통해 opensearch로 넘김

Spring boot 4.0.2

Opensearch 3.6.1

Fluent-bit 2.2.2

 

1. pom.xml

<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>8.1</version>
</dependency>

 

2. logback.xml

<appender name="JSON_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${LOG_PATH}/${WORKER_NAME}/${APP_NAME}.json</file>

        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <fileNamePattern>${LOG_PATH}/${WORKER_NAME}/archive/${APP_NAME}.%d{yyyy-MM-dd}.%i.json.gz</fileNamePattern>
            <maxFileSize>100MB</maxFileSize>
            <maxHistory>30</maxHistory>
            <totalSizeCap>5GB</totalSizeCap>
        </rollingPolicy>

        <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
            <providers>
                <timestamp>
                    <fieldName>@timestamp</fieldName>
                    <pattern>yyyy-MM-dd'T'HH:mm:ss.SSSXXX</pattern>
                    <timeZone>Asia/Seoul</timeZone>
                </timestamp>
                <threadName>
                    <fieldName>thread</fieldName>
                </threadName>
                <logLevel>
                    <fieldName>level</fieldName>
                </logLevel>
                <loggerName>
                    <fieldName>logger</fieldName>
                </loggerName>
                <message>
                    <fieldName>message</fieldName>
                </message>
                <mdc>
                    <includeMdcKeyName>requestId</includeMdcKeyName>
                    <includeMdcKeyName>userId</includeMdcKeyName>
                    <includeMdcKeyName>requestUri</includeMdcKeyName>
                    <includeMdcKeyName>requestIp</includeMdcKeyName>
                </mdc>
                <stackTrace>
                    <fieldName>stack_trace</fieldName>
                </stackTrace>
                <globalCustomFields>
                    <customFields>{"service":"${APP_NAME}","worker_name":"${WORKER_NAME}"}</customFields>
                </globalCustomFields>
            </providers>
        </encoder>
    </appender>

 

일반 file appender가 먼저 출력되는 경우 json appender로 로그가 출력되지 않을 수 있음. 어차피 opensearch로 확인하기 떄문에 file appender는 삭제했음

MDC에 저장한 값도 json 로그로 출력할 수 있음. 나중에 opensearch-dashborads에서 대시보드 만들어서 활용가능

 

3. opensearch 설치

 

2.12+에서는 설치 전에 OPENSEARCH_INITIAL_ADMIN_PASSWORD를 넘겨야 demo security가 정상 설치된다

sudo curl -SL https://artifacts.opensearch.org/releases/bundle/opensearch/3.x/opensearch-3.x.repo \
  -o /etc/yum.repos.d/opensearch-3.x.repo

sudo yum clean all
sudo yum repolist
sudo yum list opensearch --showduplicates

# 신규 설치(2.12+)는 관리자 비밀번호 필수
sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD='강한비밀번호' yum install -y opensearch

sudo systemctl enable opensearch
sudo systemctl start opensearch
sudo systemctl status opensearch

# /etc/opensearch/opensearch.yml

path.data: /home/opensearch/data
path.logs: /home/opensearch/log
network.host: 0.0.0.0
discovery.type: single-node

# demo security 자동 추가됨
plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
transport.ssl.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn: ['CN=kirk,OU=client,O=client,L=test,C=de']
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [all_access, security_rest_api_access]
plugins.security.system_indices.enabled: true
node.max_local_storage_nodes: 3

 

4. opensearch dashboard 설치

sudo curl -SL https://artifacts.opensearch.org/releases/bundle/opensearch-dashboards/3.x/opensearch-dashboards-3.x.repo \
  -o /etc/yum.repos.d/opensearch-dashboards-3.x.repo

sudo yum clean all
sudo yum repolist
sudo yum install -y opensearch-dashboards

sudo systemctl enable opensearch-dashboards
sudo systemctl start opensearch-dashboards
sudo systemctl status opensearch-dashboards

# /etc/opensearch-dashboards/opensearch_dashboards.yml
server.host: 0.0.0.0

opensearch.hosts: ["https://localhost:9200"]
opensearch.username: "kibanaserver"
opensearch.password: "kibanaserver"

opensearch.ssl.verificationMode: certificate
opensearch.ssl.certificateAuthorities: ["/etc/opensearch-dashboards/root-ca.pem"]

opensearch.requestHeadersAllowlist: ["authorization", "securitytenant"]

opensearch_security.multitenancy.enabled: true
opensearch_security.readonly_mode.roles: ["kibana_read_only"]

# Dashboards 자체는 일단 HTTP로 열기
server.ssl.enabled: false
opensearch_security.cookie.secure: false

 

opensearch 폴더에 있는 root-ca.pem 파일을 권한 오류(FATAL  Error: EACCES: permission denied, open '/etc/opensearch/root-ca.pem' ) 때문에 그냥 opensearch-dashboards 폴더에 넣었음

 

5. fluent-bit 설치

sudo yum install -y fluent-bit
sudo systemctl enable fluent-bit

[SERVICE]
    flush        1
    daemon       Off
    log_level    info
    parsers_file parsers.conf
    plugins_file plugins.conf
    http_server  Off
    http_listen  0.0.0.0
    http_port    2020
    storage.metrics on

[INPUT]
    Name                tail
    Path                /home/fluent/app-logs/app_name/*.json
    Parser              json
    Tag                 spring.app_name
    DB                  /home/fluent/log/app_name.db
    Refresh_Interval    1
    Read_From_Head      true

[INPUT]
    Name                tail
    Path                /home/fluent/app-logs/app_name/*.json
    Parser              json
    Tag                 spring.app_name
    DB                  /home/fluent/log/app_name.db
    Refresh_Interval    1
    Read_From_Head      true

[OUTPUT]
    Name          opensearch
    Match         spring.app_name*
    Host          ip
    Port          9200
    HTTP_User     admin
    HTTP_Passwd   password
    Index         spring-app_name
    tls           on
    tls.verify    off
    Suppress_Type_Name  On
    Logstash_Format     On
    Logstash_Prefix     spring-app_name
    Logstash_DateFormat %Y.%m.%d

 

fluent-bit[2196372]: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400} 에러 발생

이 에러는 Fluent Bit가 Bulk 메타데이터에 _type를 넣어서 보내고 있는데, OpenSearch 2.0 이상은 mapping type을 더 이상 지원하지 않기 때문. Fluent Bit의 suppress_type_name 기본값이 Off이고, OpenSearch 2.0+에서는 이 값을 On으로 설정해야 한다

 

개발서버도 이중화되어 있어서 하나의 인덱스로 받게금 설정했는데 각각 받고 싶으면 output 추가해야한다.

[OUTPUT]
    Name   opensearch
    Match  spring.app_name1
    Index  spring-app_name1

[OUTPUT]
    Name   opensearch
    Match  spring.app_name2
    Index  spring-app_name2

 

6. opensearch index policy 추가

 

14일 후 자동삭제

PUT _plugins/_ism/policies/spring-app_name-14d-delete
{
  "policy": {
    "description": "Delete spring-app_name indices after 14 days",
    "default_state": "hot",
    "schema_version": 1,
    "states": [
      {
        "name": "hot",
        "actions": [],
        "transitions": [
          {
            "state_name": "delete",
            "conditions": {
              "min_index_age": "14d"
            }
          }
        ]
      },
      {
        "name": "delete",
        "actions": [
          {
            "delete": {}
          }
        ],
        "transitions": []
      }
    ],
    "ism_template": {
      "index_patterns": ["spring-app_name-*"],
      "priority": 100
    }
  }
}