Plugins

Currently available Google Cloud Platform plugins for the gordon and gordon-janitor services.

Attention

These plugins are internal modules for the core gordon and gordon-janitor logic. No other use cases are expected.

Todo

Add prose documentation for how to implement a plugin.

Gordon Service

Enricher

Client module to enrich an event message with any missing information (such as IP addresses to a new hostname) and to generate the desired record(s) (e.g. A or CNAME records). Once an event message is done (either successfully enriched or met with errors along the way), it will be placed into the appropriate channel, either the success_channel or error_channel to be further handled by the gordon core system.

Attention

The enricher client is an internal module for the core gordon logic. No other use cases are expected.

class gordon_gcp.GCEEnricher(config, metrics, http_client, dns_client, **kwargs)

Get needed instance information from Google Compute Engine.

Parameters:

• config (dict) – configuration relevant to Compute Engine.
• metrics (obj) – IMetricRelay implementation.
• http_client (AIOConnection) – client for interacting with the GCE API.

handle_message(event_message)

Enrich message with extra context and send it to the publisher.

When a message is successfully processed, it is passed to the self.success_channel. However, if there is a problem during processing, the message is passed to the self.error_channel.

Parameters:event_message (GEventMessage) – message requiring additional information.

Event Consumer

Client module to consume Google Cloud Pub/Sub messages and create an event message to be passed around the gordon core system and its plugins. Once an event message is created, it will be placed into the success_channel to be further handled by the gordon core system. While being processed by the other plugin(s), the consumer will continuously extend the message’s ack deadline. When an event message is done (either successfully published or met with errors) the consumer will then ack the message in Pub/Sub to signify that the work is complete.

Attention

The event consumer client is an internal module for the core gordon logic. No other use cases are expected.

class gordon_gcp.GPSEventConsumer(config, success_channel, error_channel, metrics, subscriber, flow_control, validator, parser, loop, **kwargs)

Consume messages from Google Cloud Pub/Sub.

Pub/Sub messages are continually consumed via google-cloud-python’s pubsub module using gRPC. Every consumed message will create an asyncio task that handles the message schema validation, creation of a GEventMessage instance (event_msg), and the forwarding on for further processing via the self.success_channel. The pubsub module handles the message ack deadline extension behind the scenes. Once the event_msg is done processing, gordon’s core routing system will submit it back to this consumer to be ack’ed via the handle_method method used for pubsub cleanup.

Parameters:

• config (dict) – configuration relevant to Cloud Pub/Sub.
• success_channel (asyncio.Queue) – a sink for successfully processed interfaces.IEventMessages.
• error_channel (asyncio.Queue) – a sink for interfaces.IEventMessages that were not processed due to problems.

handle_message(event_msg)

Ack Pub/Sub message and update event message history.

Parameters:event_msg (GEventMessage) – message to clean up

run()

Start consuming messages from Google Pub/Sub.

Once a Pub/Sub message is validated, a GEventMessage is created for the message, then passed to the self.success_channel to be handled by an IEnricher plugin.

GDNS Publisher

Client module to publish DNS records from an event message. Once an event message is done (either successfully published, or met with errors along the way), it will placed into the appropriate channel, either the success_channel` or error_channel to be further handled by the gordon core system.

Attention

The publisher client is an internal module for the core gordon logic. No other use cases are expected.

class gordon_gcp.GDNSPublisher(config, metrics, dns_client, **kwargs)

Publish records to Google Cloud DNS.

Parameters:

• config (dict) – Configuration relevant to Cloud DNS.
• success_channel (asyncio.Queue) – A sink for successfully processed interfaces.IEventMessages.
• error_channel (asyncio.Queue) – A sink for interfaces.IEventMessages that were not processed due to problems.
• dns_client (gdns.GDNSClient) – A Google DNS HTTP connection class.

handle_message(event_msg)

Publish changes extracted from the event message.

Parameters:event_msg (event_consumer.GEventMessage) – Contains the changes to publish. Raises:InvalidDNSZoneInMessageError if the DNS zone of a resource record – does not match our configured zone.

Gordon Janitor

Reconciler

Module to compare desired record sets produced from a Resource Authority (i.e. GCEInstanceAuthority for Google Compute Engine) and actual record sets from Google Cloud DNS, then publish corrective messages to the internal changes_channel if there are differences.

This client makes use of the asynchronous DNS client as defined in GDNSClient, and therefore must use service account/JWT authentication (for now).

See Janitor Configuration for the required Google DNS configuration.

Attention

This reconciler client is an internal module for the core janitor logic. No other use cases are expected.

To use:

import asyncio
import gordon_gcp

config = {
    'keyfile': '/path/to/keyfile.json',
    'project': 'a-dns-project'
}
rrset_chnl = asyncio.Queue()
changes_chnl = asyncio.Queue()

reconciler = gordon_gcp.GDNSReconciler(
    config, rrset_chnl, changes_chnl)

loop = asyncio.get_event_loop()
try:
    loop.run_until_complete(reconciler.start())
finally:
    loop.close()

The keyfile is optional. If not provided the default service account will be used.

class gordon_gcp.GDNSReconciler(config, metrics, dns_client, rrset_channel=None, changes_channel=None, **kw)

Validate current records in DNS against desired source of truth.

GDNSReconciler will create a change message for the configured publisher client plugin to consume if there is a discrepancy between records in Google Cloud DNS and the desired state.

Once validation is done, the Reconciler will emit a None message to the changes_channel queue, signalling a Publisher client (e.g. GPubsubPublisher) to publish the message to a pub/sub to which Gordon subscribes.

Parameters:

• config (dict) – Google Cloud DNS-related configuration.
• metrics (obj) – IMetricRelay implementation.
• dns_client (GDNSClient) – Client to interact with Google Cloud DNS API.
• rrset_channel (asyncio.Queue) – Queue from which to consume record set messages to validate.
• changes_channel (asyncio.Queue) – Queue to publish message to make corrections to Cloud DNS.

cleanup()

Clean up & notify changes_channel of no more messages.

This method collects all tasks that this particular class initiated, and will cancel them if they don’t complete within the configured timeout period.

Once all tasks are done, None is added to the changes_channel to signify that it has no more work to process. Then the HTTP session attached to the dns_client is properly closed.

static create_rrset_set(zone, rrsets, source=None)

Create a set of ResourceRecordSets excluding SOA and zone’s NS.

Parameters:

• zone (str) – zone of the rrsets, for NS record exclusion.
• rrsets (list(dict)) – collection of dict representation of RRSets.
• source (str) – (optional) source to add to the rrset

Returns:

set of ResourceRecordSet

publish_change_messages(desired_rrsets, action='additions')

Publish change messages to the changes_channel.

NOTE: Only ‘additions’ are currently supported. ‘deletions’ may be supported in the future.

Parameters:

• desired_rrsets (list(ResourceRecordSet)) – Desired record sets that are not in Google Cloud DNS.
• action (str) – (optional) action for these corrective messages. Defaults to 'additions'.

run()

Publish necessary DNS changes to the changes_channel.

Consumes zone/rrset-list messages from rrset_channel, compares them to the current records, and publishes the changes. Once None is received from the channel, emits a final None message to the changes_channel.

validate_rrsets_by_zone(zone, rrsets)

Given a zone, validate current versus desired rrsets.

Returns lists of missing rrsets (in desired but not in current) and extra rrsets (in current but not in desired). Extra rrsets that are the result of updates in the desired list will not be returned, and root SOA/NS comparisons are skipped.

Parameters:

• zone (str) – zone to query Google Cloud DNS API.
• rrsets (list) – desired record sets to which to compare the Cloud DNS API’s response.

Returns:

The missing and extra rrset sets.

Return type:

tuple[set(rrset), set(rrset)]

GPubSub Publisher

Client module to publish any required DNS changes initiated from GDNSReconciler to Google Cloud Pub/Sub. The consumer of these messages is the Gordon service.

This client wraps around google-cloud-pubsub using grpc rather than inheriting from AIOConnection.

Attention

This publisher client is an internal module for the core janitor logic. No other use cases are expected.

To use:

import asyncio
import gordon_gcp

config = {
    'keyfile': '/path/to/keyfile.json',
    'project': 'a-dns-project',
    'topic': 'a-topic',
}
changes_channel = asyncio.Queue()

publisher = gordon_gcp.get_gpubsub_publisher(
    config, changes_channel)

loop = asyncio.get_event_loop()
try:
    loop.run_until_complete(publisher.start())
finally:
    loop.close()

The keyfile is optional. If not provided the default service account will be used.

class gordon_gcp.GPubsubPublisher(config, metrics, publisher, changes_channel=None, **kw)

Client to publish change messages to Google Pub/Sub.

Parameters:

• config (dict) – Google Cloud Pub/Sub-related configuration, ex. ‘projects/test-example/topics/a-topic’.
• publisher (google.cloud.pubsub_v1.publisher.client.Client) – client to interface with Google Pub/Sub API.
• metrics (obj) – IMetricRelay implementation.
• changes_channel (asyncio.Queue) – queue to publish message to make corrections to Cloud DNS.

cleanup()

Clean up outstanding tasks and emit final logs + metrics.

This method collects all tasks that this particular class initiated, and will cancel them if they don’t complete within the configured timeout period.

publish(message)

Publish received change message to Google Pub/Sub.

Parameters:message (dict) – change message received from the changes_channel to emit.

run()

Start consuming from changes_channel.

Once None is received from the channel, finish processing records and clean up any outstanding tasks.

Authority

A GCEAuthority retrieves a list of all instances in all projects that it has access to, and which belong to the configured zone. For every project, it will create a message containing domain record information and put it into the rrset channel. Projects can be filtered by ‘project name’. Instances can be filtered by tags and metadata.

To use:

import asyncio

from gordon_gcp.plugins import janitor

async def run():
    rrset_channel = asyncio.queue()
    authority = janitor.get_authority(config, rrset_channel)
    await authority.start()
    msg = await rrset_channel.get()
    print(msg)

loop = asyncio.get_event_loop()
loop.run_until_complete(run())
# prints: {'zone': 'example.com', 'resourceRecords': [...]}

class gordon_gcp.GCEAuthority(config, metrics, crm_client, gce_client, rrset_channel=None, **kwargs)

Gather instance data from GCE.

Parameters:

• config (dict) – plugin-specific configuration.
• metrics (obj) – IMetricRelay implementation.
• crm_client (GCRMClient) – client used to fetch GCE projects.
• gce_client (GCEClient) – client used to fetch instances for a project.
• rrset_channel (asyncio.Queue) – channel to send resource record messages to.

cleanup()

Clean up after a run.

run()

Batch instance data and send it to the self.rrset_channel.