Schemas for Event Consuming¶

Definitions¶

This package defines two types of schemas for consuming events by the Google Pub/Sub Event Consumer plugin: Google Audit Log Message and a General Event Message.

Google Audit Log Message¶

The Google Audit Log Message schema is based off of Google’s Audit Log datatype for Cloud Audit Logging. Documentation on setting up the exporting of Google Cloud audit logs to Google Pub/Sub (a.k.a. a “sink”) can be found here. You may need to refine the export to include the following filters:

resource.type=gce_instance
protoPayload."@type"="type.googleapis.com/google.cloud.audit.AuditLog"

Schema Definition¶

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "type": "object",
    "id": "https://github.com/spotify/gordon-gcp/blob/master/gordon_gcp/schemas/audit-log.schema.json",
    "title": "Google Audit Log Messages",
    "description": "Schema for parsing Google Audit Log Messages",
    "required": ["protoPayload", "logName"],
    "properties": {
        "protoPayload": {
            "type": "object",
            "properties": {
                "methodName": {
                    "type": "string",
                    "description": "Action performed on a GCP",
                    "enum": [
                        "v1.compute.instances.insert",
                        "v1.compute.instances.delete"
                    ]
                },
                "resourceName": {
                    "type": "string",
                    "description": "",
                    "examples": [
                        "projects/a-project-id/zones/a-zone-name/instances/an-instance-name"
                    ],
                    "pattern": "^projects/([0-9]+|[a-z][a-z\\-0-9]{5,29})/zones/[a-z\\-0-9]+/instances/[a-z\\-0-9]+"
                }
            },
            "required": ["methodName", "resourceName"]
        },
        "logName": {
            "type": "string",
            "description": "Name of audit log",
            "examples": [
                "projects/a-project-id/logs/cloudaudit.googleapis.com%2Factivity"
            ],
            "pattern": "^projects/[a-z][a-z\\-0-9]{5,29}/logs/cloudaudit.googleapis.com%2Factivity"
        },
        "receiveTimestamp": {
            "type": "string",
            "description": "Time the log entry was received by Stackdriver Logging",
            "examples": [
                "2018-01-01T23:13:45.123456789Z"
            ]
        },
        "timestamp": {
            "type": "string",
            "description": "Time the operation status was reported",
            "examples": [
                "2018-01-01T23:13:45.123456789Z"
            ]
        }
    }
}

Examples¶

Audit Log Message with an `operation.first` key:¶

{
  "insertId": "123456789101112",
  "logName": "projects/some-project/logs/cloudaudit.googleapis.com%2Factivity",
  "operation": {
    "first": true,
    "id": "operation-1234-5678-9101-1121",
    "producer": "compute.googleapis.com"
  },
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "authenticationInfo": {
      "principalEmail": "not-real@cloudservices.gserviceaccount.com"
    },
    "authorizationInfo": [
      {
        "granted": true,
        "permission": "compute.instances.create"
      }
    ],
    "methodName": "v1.compute.instances.insert",
    "request": {
      "@type": "compute.googleapis.com/compute.instances.insert"
    },
    "requestMetadata": {
      "callerSuppliedUserAgent": "Managed Infrastructure Mixer Client"
    },
    "resourceName": "projects/123456789101/zones/us-central1-c/instances/an-instance-name-b34c",
    "response": {
      "@type": "compute.googleapis.com/operation",
      "id": "234567891011121314",
      "insertTime": "2017-12-04T12:13:44.785-08:00",
      "name": "operation-1234-5678-9101-1121",
      "operationType": "insert",
      "progress": "0",
      "selfLink": "https://www.googleapis.com/compute/v1/projects/some-project/zones/us-central1-c/operations/operation-1234-5678-9101-1121",
      "status": "PENDING",
      "targetId": "123456789101213141",
      "targetLink": "https://www.googleapis.com/compute/v1/projects/some-project/zones/us-central1-c/instances/an-instance-name-b34c",
      "zone": "https://www.googleapis.com/compute/v1/projects/some-project/zones/us-central1-c"
    },
    "serviceName": "compute.googleapis.com"
  },
  "receiveTimestamp": "2017-12-04T20:13:45.494149958Z",
  "resource": {
    "labels": {
      "instance_id": "123456789101213141",
      "project_id": "some-project",
      "zone": "us-central1-c"
    },
    "type": "gce_instance"
  },
  "severity": "NOTICE",
  "timestamp": "2017-12-04T20:13:44.181Z"
}

Audit Log Message with an `operation.last` key:¶

{
    "insertId": "123456789101112",
    "logName": "projects/some-project/logs/cloudaudit.googleapis.com%2Factivity",
    "operation": {
        "id": "operation-1234-5678-9101-1121",
        "last": true,
        "producer": "compute.googleapis.com"
    },
    "protoPayload": {
        "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
        "authenticationInfo": {},
        "methodName": "v1.compute.instances.insert",
        "requestMetadata": {
            "callerSuppliedUserAgent": "Managed Infrastructure Mixer Client"
        },
        "resourceName": "projects/123456789101/zones/us-central1-c/instances/an-instance-name-b45c",
        "serviceName": "compute.googleapis.com"
    },
    "receiveTimestamp": "2017-12-04T20:13:51.414016721Z",
    "resource": {
        "labels": {
            "instance_id": "123456789101213141",
            "project_id": "some-project",
            "zone": "us-central1-c"
        },
        "type": "gce_instance"
    },
    "severity": "NOTICE",
    "timestamp": "2017-12-04T20:13:50.717Z"
}

Audit Log Message with no `operation` object:¶

{
    "insertId": "123456789101112",
    "logName": "projects/a-project/logs/cloudaudit.googleapis.com%2Factivity",
    "protoPayload": {
        "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
        "authenticationInfo": {},
        "methodName": "v1.compute.instances.insert",
        "requestMetadata": {
            "callerSuppliedUserAgent": "Managed Infrastructure Mixer Client"
        },
        "resourceName": "projects/123456789101/zones/us-central1-c/instances/an-instance-name-a123b",
        "serviceName": "compute.googleapis.com"
    },
    "receiveTimestamp": "2017-12-04T20:13:51.414016721Z",
    "resource": {
        "labels": {
            "instance_id": "123456789101213141",
            "project_id": "a-project",
            "zone": "us-central1-c"
        },
        "type": "gce_instance"
    },
    "severity": "NOTICE",
    "timestamp": "2017-12-04T20:13:50.717Z"
}

General Event Message¶

The General Event Message schema is meant for any event that may not come from Google’s audit log sink. For instance, one may need to add/update/delete a manual record (i.e. marketing-friendly CNAME s). Or there may be a reconciliation process between the current state of the world and what is reflected in DNS.

Schema Definition¶

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "type": "object",
    "id": "https://github.com/spotify/gordon-gcp/blob/master/gordon_gcp/schemas/event.schema.json",
    "title": "Generic Event Messages",
    "description": "Schema for parsing generic event messages",
    "required": ["action", "timestamp", "resourceRecords"],
    "properties": {
        "action": {
            "description": "Add or delete resource record sets (rrdatas) listed",
            "type": "string",
            "enum": ["additions", "deletions"]
        },
        "timestamp": {
            "type": "string",
            "description": "Time the message was emitted",
            "examples": [
                "2018-01-01T23:13:45.123456789Z"
            ]
        },
        "resourceRecords": {
            "description": "DNS resource records upon which to act",
            "type": "object",
            "required": ["name", "rrdatas", "type"],
            "properties": {
                "name": {
                    "description": "Record name",
                    "type": "string",
                    "examples": [
                        "www.example.com.",
                        "some-fqdn.subdomain.example.com."
                    ]
                },
                "rrdatas": {
                    "description": "Resource record(s) that `name` point to",
                    "type": "array",
                    "minItems": 1,
                    "items": {
                        "type": "string"
                    },
                    "uniqueItems": true,
                    "examples": [
                        "198.1.1.50",
                        "example.com.",
                        "some text value"
                    ]
                },
                "ttl": {
                    "description": "TTL in seconds; default 300 seconds",
                    "type": "number",
                    "maximum": 86400,
                    "minimum": 60,
                    "default": 300
                },
                "type": {
                    "description": "Type of resource record(s) as supported by Google Cloud DNS",
                    "type": "string",
                    "enum": ["A", "AAAA", "CNAME", "MX", "NS", "PTR", "SOA", "SRV", "TXT"]
                }
            }
        }
    }
}

Examples¶

Event for an `A` record:¶

{
    "resourceRecords": {
        "name": "lynntest-a-1234.foo.spotify.net.",
        "rrdatas": [
            "10.1.2.3"
        ],
        "ttl": 300,
        "type": "A"
    },
    "timestamp": "2017-12-04T20:13:51.414016721Z",
    "action": "additions"
}

Event for an `CNAME` record:¶

{
    "action": "additions",
    "timestamp": "2017-12-04T20:13:51.414016721Z",
    "resourceRecords": {
        "name": "lynntest-a-1234.spotify.net.",
        "rrdatas": [
            "lynntest-a-1234.foo.spotify.net."
        ],
        "ttl": 300,
        "type": "CNAME"
    }
}

Event for an `NS` record:¶

{
    "action": "additions",
    "timestamp": "2017-12-04T20:13:51.414016721Z",
    "resourceRecords": {
        "name": "gordontest.spotify.net.",
        "rrdatas": [
            "ns-cloud-c1.googledomains.com.",
            "ns-cloud-c2.googledomains.com.",
            "ns-cloud-c3.googledomains.com.",
            "ns-cloud-c4.googledomains.com."
        ],
        "ttl": 21600,
        "type": "SOA"
    }
}

Validating¶

Module to load and validate GCP-related JSON schemas.

Schema file discovery is based on any JSON file in the gordon_gcp/schema/schemas/ directory.

For more information on this plugin’s schemas, see Schemas for Event Consuming.

Warning

The following documentation may change since the calling/using of the this module is not yet incorporated into the plugin logic.

Schemas are loaded once at service startup. A JSON message/object is validated against a given loaded schema upon receipt of message/object. The schema to validate against is determined by the topic from which the PubSub message is pulled.

To use:

>>> from gordon_gcp.schema import validate
>>> validator = validate.MessageValidator()
>>> validator.schemas
{
    'event': {
        'title': 'Generic Event Message',
        'type': 'object',
        'required': ...},
    'audit-log': {
        'title': 'Google Audit Log Message',
        'type': 'object',
        'required': ...}
}
>>> example_message = {'foo': 'bar'}
>>> validator.validate(example_message, 'event')

class gordon_gcp.MessageValidator[source]¶

Load packaged JSON schemas and validate a given JSON message.

schemas¶: dict – schema name based on filename mapped to its loaded JSON schema.

Raises:	`GCPGordonError` – if unable to find or load schemas.

validate(message, schema_name)[source]¶

Validate a message given a schema.

Parameters:	message (dict) – Loaded JSON of pulled message from Google PubSub. schema_name (str) – Name of schema to validate `message` against. `schema_name` will be used to look up schema from `MessageValidator.schemas` dict
Raises:	`InvalidMessageError` – if message is invalid against the given schema. `InvalidMessageError` – if given schema name can not be found.

Parsing¶

Module to parse loaded JSON messages according to GCP-related schemas and return only the actionable data.

To use:

>>> import json
>>> from gordon_gcp.schema import parse
>>> parser = parse.MessageParser()
>>> # using an example file
>>> exp = 'gordon_gcp/schema/examples/audit-log.last-operation.json'
>>> with open(exp, 'r') as f:
...   data = json.load(f)
>>> message = parser.parse(data, 'audit-log')
>>> message
{
    'action': 'additions',
    'resourceName': 'projects/.../instances/an-instance-name-b45c',
    'resourceRecords': []
}

class gordon_gcp.MessageParser[source]¶

Parse a message provided a given GCP schema.

parse(message, schema)[source]¶

Parse message according to schema.

message should already be validated against the given schema. See Schema Definition for more information.

Parameters:	message (dict) – message data to parse. schema (str) – valid message schema.
Returns:	parsed message
Return type:	(dict)

Schemas for Event Consuming¶

Definitions¶

Google Audit Log Message¶

Schema Definition¶

Examples¶

Audit Log Message with an `operation.first` key:¶

Audit Log Message with an `operation.last` key:¶

Audit Log Message with no `operation` object:¶

General Event Message¶

Schema Definition¶

Examples¶

Event for an `A` record:¶

Event for an `CNAME` record:¶

Event for an `NS` record:¶

Validating¶

Parsing¶

Table Of Contents

Related Topics

Schemas for Event Consuming¶

Definitions¶

Google Audit Log Message¶

Schema Definition¶

Examples¶

Audit Log Message with an operation.first key:¶

Audit Log Message with an operation.last key:¶

Audit Log Message with no operation object:¶

General Event Message¶

Schema Definition¶

Examples¶

Event for an A record:¶

Event for an CNAME record:¶

Event for an NS record:¶

Validating¶

Parsing¶

Audit Log Message with an `operation.first` key:¶

Audit Log Message with an `operation.last` key:¶

Audit Log Message with no `operation` object:¶

Event for an `A` record:¶

Event for an `CNAME` record:¶

Event for an `NS` record:¶