Schemas for Event Consuming

Definitions

This package defines two types of schemas for consuming events by the Google Pub/Sub Event Consumer plugin: Google Audit Log Message and a General Event Message.

Google Audit Log Message

The Google Audit Log Message schema is based off of Google’s Audit Log datatype for Cloud Audit Logging. Documentation on setting up the exporting of Google Cloud audit logs to Google Pub/Sub (a.k.a. a “sink”) can be found here. You may need to refine the export to include the following filters:

resource.type=gce_instance
protoPayload."@type"="type.googleapis.com/google.cloud.audit.AuditLog"

Schema Definition

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "type": "object",
    "id": "https://github.com/spotify/gordon-gcp/blob/master/gordon_gcp/schemas/audit-log.schema.json",
    "title": "Google Audit Log Messages",
    "description": "Schema for parsing Google Audit Log Messages",
    "required": ["protoPayload", "logName"],
    "properties": {
        "protoPayload": {
            "type": "object",
            "properties": {
                "methodName": {
                    "type": "string",
                    "description": "Action performed on a GCP",
                    "enum": [
                        "v1.compute.instances.insert",
                        "v1.compute.instances.delete"
                    ]
                },
                "resourceName": {
                    "type": "string",
                    "description": "",
                    "examples": [
                        "projects/a-project-id/zones/a-zone-name/instances/an-instance-name"
                    ],
                    "pattern": "^projects/[0-9]+/zones/[a-z\\-0-9]+/instances/[a-z\\-0-9]+"
                }
            },
            "required": ["methodName", "resourceName"]
        },
        "logName": {
            "type": "string",
            "description": "Name of audit log",
            "examples": [
                "projects/a-project-id/logs/cloudaudit.googleapis.com%2Factivity"
            ],
            "pattern": "^projects/[a-z][a-z\\-0-9]{5,29}/logs/cloudaudit.googleapis.com%2Factivity"
        },
        "receiveTimestamp": {
            "type": "string",
            "description": "Time the log entry was received by Stackdriver Logging",
            "examples": [
                "2018-01-01T23:13:45.123456789Z"
            ]
        },
        "timestamp": {
            "type": "string",
            "description": "Time the operation status was reported",
            "examples": [
                "2018-01-01T23:13:45.123456789Z"
            ]
        }
    }
}

Examples

Audit Log Message with an operation.first key:
{
  "insertId": "123456789101112",
  "logName": "projects/some-project/logs/cloudaudit.googleapis.com%2Factivity",
  "operation": {
    "first": true,
    "id": "operation-1234-5678-9101-1121",
    "producer": "compute.googleapis.com"
  },
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "authenticationInfo": {
      "principalEmail": "not-real@cloudservices.gserviceaccount.com"
    },
    "authorizationInfo": [
      {
        "granted": true,
        "permission": "compute.instances.create"
      }
    ],
    "methodName": "v1.compute.instances.insert",
    "request": {
      "@type": "compute.googleapis.com/compute.instances.insert"
    },
    "requestMetadata": {
      "callerSuppliedUserAgent": "Managed Infrastructure Mixer Client"
    },
    "resourceName": "projects/123456789101/zones/us-central1-c/instances/an-instance-name-b34c",
    "response": {
      "@type": "compute.googleapis.com/operation",
      "id": "234567891011121314",
      "insertTime": "2017-12-04T12:13:44.785-08:00",
      "name": "operation-1234-5678-9101-1121",
      "operationType": "insert",
      "progress": "0",
      "selfLink": "https://www.googleapis.com/compute/v1/projects/some-project/zones/us-central1-c/operations/operation-1234-5678-9101-1121",
      "status": "PENDING",
      "targetId": "123456789101213141",
      "targetLink": "https://www.googleapis.com/compute/v1/projects/some-project/zones/us-central1-c/instances/an-instance-name-b34c",
      "zone": "https://www.googleapis.com/compute/v1/projects/some-project/zones/us-central1-c"
    },
    "serviceName": "compute.googleapis.com"
  },
  "receiveTimestamp": "2017-12-04T20:13:45.494149958Z",
  "resource": {
    "labels": {
      "instance_id": "123456789101213141",
      "project_id": "some-project",
      "zone": "us-central1-c"
    },
    "type": "gce_instance"
  },
  "severity": "NOTICE",
  "timestamp": "2017-12-04T20:13:44.181Z"
}
Audit Log Message with an operation.last key:
{
    "insertId": "123456789101112",
    "logName": "projects/some-project/logs/cloudaudit.googleapis.com%2Factivity",
    "operation": {
        "id": "operation-1234-5678-9101-1121",
        "last": true,
        "producer": "compute.googleapis.com"
    },
    "protoPayload": {
        "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
        "authenticationInfo": {},
        "methodName": "v1.compute.instances.insert",
        "requestMetadata": {
            "callerSuppliedUserAgent": "Managed Infrastructure Mixer Client"
        },
        "resourceName": "projects/123456789101/zones/us-central1-c/instances/an-instance-name-b45c",
        "serviceName": "compute.googleapis.com"
    },
    "receiveTimestamp": "2017-12-04T20:13:51.414016721Z",
    "resource": {
        "labels": {
            "instance_id": "123456789101213141",
            "project_id": "some-project",
            "zone": "us-central1-c"
        },
        "type": "gce_instance"
    },
    "severity": "NOTICE",
    "timestamp": "2017-12-04T20:13:50.717Z"
}
Audit Log Message with no operation object:
{
    "insertId": "123456789101112",
    "logName": "projects/a-project/logs/cloudaudit.googleapis.com%2Factivity",
    "protoPayload": {
        "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
        "authenticationInfo": {},
        "methodName": "v1.compute.instances.insert",
        "requestMetadata": {
            "callerSuppliedUserAgent": "Managed Infrastructure Mixer Client"
        },
        "resourceName": "projects/123456789101/zones/us-central1-c/instances/an-instance-name-a123b",
        "serviceName": "compute.googleapis.com"
    },
    "receiveTimestamp": "2017-12-04T20:13:51.414016721Z",
    "resource": {
        "labels": {
            "instance_id": "123456789101213141",
            "project_id": "a-project",
            "zone": "us-central1-c"
        },
        "type": "gce_instance"
    },
    "severity": "NOTICE",
    "timestamp": "2017-12-04T20:13:50.717Z"
}

General Event Message

The General Event Message schema is meant for any event that may not come from Google’s audit log sink. For instance, one may need to add/update/delete a manual record (i.e. marketing-friendly CNAME s). Or there may be a reconciliation process between the current state of the world and what is reflected in DNS.

Schema Definition

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "type": "object",
    "id": "https://github.com/spotify/gordon-gcp/blob/master/gordon_gcp/schemas/event.schema.json",
    "title": "Generic Event Messages",
    "description": "Schema for parsing generic event messages",
    "required": ["action", "timestamp", "resourceRecords"],
    "properties": {
        "action": {
            "description": "Add or delete resource record sets (rrdatas) listed",
            "type": "string",
            "enum": ["additions", "deletions"]
        },
        "timestamp": {
            "type": "string",
            "description": "Time the message was emitted",
            "examples": [
                "2018-01-01T23:13:45.123456789Z"
            ]
        },
        "resourceRecords": {
            "description": "DNS resource records upon which to act",
            "type": "object",
            "required": ["name", "rrdatas", "type"],
            "properties": {
                "name": {
                    "description": "Record name",
                    "type": "string",
                    "examples": [
                        "www.example.com.",
                        "some-fqdn.subdomain.example.com."
                    ]
                },
                "rrdatas": {
                    "description": "Resource record(s) that `name` point to",
                    "type": "array",
                    "minItems": 1,
                    "items": {
                        "type": "string"
                    },
                    "uniqueItems": true,
                    "examples": [
                        "198.1.1.50",
                        "example.com.",
                        "some text value"
                    ]
                },
                "ttl": {
                    "description": "TTL in seconds; default 300 seconds",
                    "type": "number",
                    "maximum": 86400,
                    "minimum": 60,
                    "default": 300
                },
                "type": {
                    "description": "Type of resource record(s) as supported by Google Cloud DNS",
                    "type": "string",
                    "enum": ["A", "AAAA", "CNAME", "MX", "NS", "PTR", "SOA", "SRV", "TXT"]
                }
            }
        }
    }
}

Examples

Event for an A record:
{
    "resourceRecords": {
        "name": "lynntest-a-1234.foo.spotify.net.",
        "rrdatas": [
            "10.1.2.3"
        ],
        "ttl": 300,
        "type": "A"
    },
    "timestamp": "2017-12-04T20:13:51.414016721Z",
    "action": "additions"
}
Event for an CNAME record:
{
    "action": "additions",
    "timestamp": "2017-12-04T20:13:51.414016721Z",
    "resourceRecords": {
        "name": "lynntest-a-1234.spotify.net.",
        "rrdatas": [
            "lynntest-a-1234.foo.spotify.net."
        ],
        "ttl": 300,
        "type": "CNAME"
    }
}
Event for an NS record:
{
    "action": "additions",
    "timestamp": "2017-12-04T20:13:51.414016721Z",
    "resourceRecords": {
        "name": "gordontest.spotify.net.",
        "rrdatas": [
            "ns-cloud-c1.googledomains.com.",
            "ns-cloud-c2.googledomains.com.",
            "ns-cloud-c3.googledomains.com.",
            "ns-cloud-c4.googledomains.com."
        ],
        "ttl": 21600,
        "type": "SOA"
    }
}

Validating

Module to load and validate GCP-related JSON schemas.

Schema file discovery is based on any JSON file in the gordon_gcp/schema/schemas/ directory.

For more information on this plugin’s schemas, see Schemas for Event Consuming.

Warning

The following documentation may change since the calling/using of the this module is not yet incorporated into the plugin logic.

Schemas are loaded once at service startup. A JSON message/object is validated against a given loaded schema upon receipt of message/object. The schema to validate against is determined by the topic from which the PubSub message is pulled.

To use:

>>> from gordon_gcp.schema import validate
>>> validator = validate.MessageValidator()
>>> validator.schemas
{
    'event': {
        'title': 'Generic Event Message',
        'type': 'object',
        'required': ...},
    'audit-log': {
        'title': 'Google Audit Log Message',
        'type': 'object',
        'required': ...}
}
>>> example_message = {'foo': 'bar'}
>>> validator.validate(example_message, 'event')
class gordon_gcp.MessageValidator[source]

Load packaged JSON schemas and validate a given JSON message.

schemas

dict – schema name based on filename mapped to its loaded JSON schema.

Raises:GCPGordonError – if unable to find or load schemas.
validate(message, schema_name)[source]

Validate a message given a schema.

Parameters:
  • message (dict) – Loaded JSON of pulled message from Google PubSub.
  • schema_name (str) – Name of schema to validate message against. schema_name will be used to look up schema from MessageValidator.schemas dict
Raises:

Parsing

Module to parse loaded JSON messages according to GCP-related schemas and return only the actionable data.

To use:

>>> import json
>>> from gordon_gcp.schema import parse
>>> parser = parse.MessageParser()
>>> # using an example file
>>> exp = 'gordon_gcp/schema/examples/audit-log.last-operation.json'
>>> with open(exp, 'r') as f:
...   data = json.load(f)
>>> message = parser.parse(data, 'audit-log')
>>> message
{
    'action': 'additions',
    'resourceName': 'projects/.../instances/an-instance-name-b45c',
    'resourceRecords': []
}
class gordon_gcp.MessageParser[source]

Parse a message provided a given GCP schema.

parse(message, schema)[source]

Parse message according to schema.

message should already be validated against the given schema. See Schema Definition for more information.

Parameters:
  • message (dict) – message data to parse.
  • schema (str) – valid message schema.
Returns:

parsed message

Return type:

(dict)