grafana-agent内置集成了process-exporter，基于/proc的文件分析结果，来收集Linux系统进程相关的指标（注意，非Linux系统开启该exporter不起作用）。

如果grafana-agent运行在container中，那么在容器的启动命令中，要做以下调整，即将宿主机的/proc目录映射到容器中相应的位置。

  1. docker run \
  2.   -v "/proc:/proc:ro" \
  3.   -v /tmp/agent:/etc/agent \
  4.   -v /path/to/config.yaml:/etc/agent-config/agent.yaml \
  5.   grafana/agent:v0.23.0 \
  6.   --config.file=/etc/agent-config/agent.yaml

注意，将/path/to/config.yaml替换成您自己相应的配置文件。

如果grafana-agent运行在Kubernetes中，那么同样的需要在manifest文件中，做如下调整，即将宿主机的/proc目录映射到容器中相应的位置。

  1. apiVersion: v1
  2. kind: Pod
  3. metadata:
  4.   name: grafana-agent
  5. spec:
  6.   containers:
  7.   - image: grafana/agent:v0.23.0
  8.     name: agent
  9.     args:
  10.     - --config.file=/etc/agent-config/agent.yaml
  11.     volumeMounts:
  12.     - name: procfs
  13.       mountPath: /proc
  14.       readOnly: true
  15.   volumes:
  16.   - name: procfs
  17.     hostPath:
  18.       path: /proc

配置并启用process_exporter

如下的配置，将会开启process_exporter，并追踪系统中的所有进程。

  1. process_exporter:
  2.   enabled: true
  3.   process_names:
  4.   - name: "{{.Comm}}"
  5.     cmdline:
  6.     - '.+'

采集的指标列表

  1. # Context switches
  2. # 上下文切换数量
  3. # Counter
  4. namedprocess_namegroup_context_switches_total
  6. # Cpu user/system usage in seconds
  7. # CPU 时间（秒）
  8. # Counter
  9. namedprocess_namegroup_cpu_seconds_total
  11. # Major page faults
  12. # 主要页缺失次数
  13. # Counter
  14. namedprocess_namegroup_major_page_faults_total
  16. # Minor page faults
  17. # 次要页缺失次数
  18. # Counter
  19. namedprocess_namegroup_minor_page_faults_total
  21. # number of bytes of memory in use
  22. # 内存占用（byte）
  23. # Gauge
  24. namedprocess_namegroup_memory_bytes
  26. # number of processes in this group
  27. # 同名进程数量
  28. # Gauge
  29. namedprocess_namegroup_num_procs
  31. # Number of processes in states Running, Sleeping, Waiting, Zombie, or Other
  32. # 同名进程状态分布
  33. # Gauge
  34. namedprocess_namegroup_states
  36. # Number of threads
  37. # 线程数量
  38. # Gauge
  39. namedprocess_namegroup_num_threads
  41. # start time in seconds since 1970/01/01 of oldest process in group
  42. # 启动时间戳
  43. # Gauge
  44. namedprocess_namegroup_oldest_start_time_seconds
  46. # number of open file descriptors for this group
  47. # 打开文件描述符数量
  48. # Gauge
  49. namedprocess_namegroup_open_filedesc
  51. # the worst (closest to 1) ratio between open fds and max fds among all procs in this group
  52. # 打开文件数 / 允许打开文件数
  53. # Gauge
  54. namedprocess_namegroup_worst_fd_ratio
  56. # number of bytes read by this group
  57. # 读数据量（byte）
  58. # Counter
  59. namedprocess_namegroup_read_bytes_total
  61. # number of bytes written by this group
  62. # 写数据量（byte）
  63. # Counter
  64. namedprocess_namegroup_write_bytes_total
  66. # Number of threads in this group waiting on each wchan
  67. # 内核wchan等待线程数量
  68. # Gauge
  69. namedprocess_namegroup_threads_wchan

process_exporter的详细配置项说明

  1.   # Enables the process_exporter integration, allowing the Agent to automatically
  2.   # collect system metrics from the host UNIX system.
  3.   [enabled: <boolean> | default = false]
  5.   # Sets an explicit value for the instance label when the integration is
  6.   # self-scraped. Overrides inferred values.
  7.   #
  8.   # The default value for this integration is inferred from the agent hostname
  9.   # and HTTP listen port, delimited by a colon.
  10.   [instance: <string>]
  12.   # Automatically collect metrics from this integration. If disabled,
  13.   # the process_exporter integration will be run but not scraped and thus not
  14.   # remote-written. Metrics for the integration will be exposed at
  15.   # /integrations/process_exporter/metrics and can be scraped by an external
  16.   # process.
  17.   [scrape_integration: <boolean> | default = <integrations_config.scrape_integrations>]
  19.   # How often should the metrics be collected? Defaults to
  20.   # prometheus.global.scrape_interval.
  21.   [scrape_interval: <duration> | default = <global_config.scrape_interval>]
  23.   # The timeout before considering the scrape a failure. Defaults to
  24.   # prometheus.global.scrape_timeout.
  25.   [scrape_timeout: <duration> | default = <global_config.scrape_timeout>]
  27.   # Allows for relabeling labels on the target.
  28.   relabel_configs:
  29.     [- <relabel_config> ... ]
  31.   # Relabel metrics coming from the integration, allowing to drop series
  32.   # from the integration that you don't care about.
  33.   metric_relabel_configs:
  34.     [ - <relabel_config> ... ]
  36.   # How frequent to truncate the WAL for this integration.
  37.   [wal_truncate_frequency: <duration> | default = "60m"]
  39.   # procfs mountpoint.
  40.   [procfs_path: <string> | default = "/proc"]
  42.   # If a proc is tracked, track with it any children that aren't a part of their
  43.   # own group.
  44.   [track_children: <boolean> | default = true]
  46.   # Report on per-threadname metrics as well.
  47.   [track_threads: <boolean> | default = true]
  49.   # Gather metrics from smaps file, which contains proportional resident memory
  50.   # size.
  51.   [gather_smaps: <boolean> | default = true]
  53.   # Recheck process names on each scrape.
  54.   [recheck_on_scrape: <boolean> | default = false]
  56.   # A collection of matching rules to use for deciding which processes to
  57.   # monitor. Each config can match multiple processes to be tracked as a single
  58.   # process "group."
  59.   process_names:
  60.     [- <process_matcher_config>]

process_matcher_config

  1. # The name to use for identifying the process group name in the metric. By
  2. # default, it uses the base path of the executable.
  3. #
  4. # The following template variables are available:
  5. #
  6. # - {{.Comm}}:      Basename of the original executable from /proc/<pid>/stat
  7. # - {{.ExeBase}}:   Basename of the executable from argv[0]
  8. # - {{.ExeFull}}:   Fully qualified path of the executable
  9. # - {{.Username}}:  Username of the effective user
  10. # - {{.Matches}}:   Map containing all regex capture groups resulting from
  11. #                   matching a process with the cmdline rule group.
  12. # - {{.PID}}:       PID of the process. Note that the PID is copied from the
  13. #                   first executable found.
  14. # - {{.StartTime}}: The start time of the process. This is useful when combined
  15. #                   with PID as PIDS get reused over time.
  16. [name: <string> | default = "{{.ExeBase}}"]
  18. # A list of strings that match the base executable name for a process, truncated
  19. # at 15 characters. It is derived from reading the second field of
  20. # /proc/<pid>/stat minus the parens.
  21. #
  22. # If any of the strings match, the process will be tracked.
  23. comm:
  24.   [- <string>]
  26. # A list of strings that match argv[0] for a process. If there are no slashes,
  27. # only the basename of argv[0] needs to match. Otherwise the name must be an
  28. # exact match. For example, "postgres" may match any postgres binary but
  29. # "/usr/local/bin/postgres" can only match a postgres at that path exactly.
  30. #
  31. # If any of the strings match, the process will be tracked.
  32. exe:
  33.   [- <string>]
  35. # A list of regular expressions applied to the argv of the process. Each
  36. # regex here must match the corresponding argv for the process to be tracked.
  37. # The first element that is matched is argv[1].
  38. #
  39. # Regex Captures are added to the .Matches map for use in the name.
  40. cmdline:
  41.   [- <string>]