Also read Logging in GCP

Cloud Monitoring collects metrics, events, and metadata from Google Cloud and applications  hosted on GCP.

GCP provides several built in metrics

What about ALERTING and NOTIFICATIONs based on metrics (thresholds)?

Sample Process Health Policy and Notification when Process Stops (for over 5 minutes threshold)

{
"displayName": "Server health",
"combiner": "OR",
"conditions": [
{
"displayName": "Process 'nginx' is not running",
"conditionThreshold": {
"filter": "select_process_count(\"has_substring(\\\"nginx\\\")\", \"www\") AND resource.type=\"gce_instance\"",
"comparison": "COMPARISON_LT",
"thresholdValue": 1,
"duration": "300s"
}
}
],
}

Monitoring Account Activity versus RESOURCE monitoring

One ADDITIONAL thing that is MONITORED is Admin Activity. It is good to conceptually think of IAM Activity and Admin Activity EVENTS as different from the RESOURCE monitoring.

Another type of metric that comes straight from the logs is a log-based metric. This is like an average (or aggregate) of  a particular log event  - e.g. - if you see a particular error type happening 3 times a day, you can create a l0g-based metric for that error - and alert if the threshold (count) crosses 3 per day.

Sending Logs to another Monitoring Service

Best Practice would entail forwarding all audit logs to a monitoring service. This ensures that you do not miss high severity or critical severity events.  You need the following gCloud commands to create SINKS for your logs.

Note that you have to provide a sink-name, a topic-name, a folder id and most importantly, the log-filter that is passed in directly to the sinks create command as an argument.

$ gcloud logging sinks create {sink-name} pubsub.googleapis.com/projects/{project-id}/topics/{topic-name} --organization={organization-id} --log-filter="protoPayload.@type=\"type.googleapis.com/google.cloud.audit.AuditLog\""
$ gcloud logging sinks create {sink-name} pubsub.googleapis.com/projects/{project-id}/topics/{topic-name} --folder={folder-id} --log-filter="protoPayload.@type=\"type.googleapis.com/google.cloud.audit.AuditLog\""

The following types of audit logs are available for Monitoring:

  • Admin Activity audit logsIncludes "admin write" operations that write metadata or configuration information.You can't disable Admin Activity audit logs.
  • Data Access audit logsIncludes "admin read" operations that read metadata or configuration information. Also includes "data read" and "data write" operations that read or write user-provided data.To receive Data Access audit logs, you must explicitly enable them.

For fuller descriptions of the audit log types, see Types of audit logs.