Plugins¶
Currently available Google Cloud Platform plugins for the gordon and gordon-janitor services.
Attention
These plugins are internal modules for the core gordon and gordon-janitor logic. No other use cases are expected.
Todo
Add prose documentation for how to implement a plugin.
Gordon Service¶
Enricher¶
Client module to enrich an event message with any missing information
(such as IP addresses to a new hostname) and to generate the desired
record(s) (e.g. A
or CNAME
records). Once an event message is
done (either successfully enriched or met with errors along the way),
it will be placed into the appropriate channel, either the
success_channel
or error_channel
to be further handled by the
gordon
core system.
Attention
The enricher client is an internal module for the core gordon logic. No other use cases are expected.
-
class
gordon_gcp.
GCEEnricher
(config, metrics, http_client, dns_client, **kwargs)[source]¶ Get needed instance information from Google Compute Engine.
Parameters: - config (dict) – configuration relevant to Compute Engine.
- metrics (obj) –
IMetricRelay
implementation. - http_client (AIOConnection) – client for interacting with the GCE API.
-
handle_message
(event_message)[source]¶ Enrich message with extra context and send it to the publisher.
When a message is successfully processed, it is passed to the
self.success_channel
. However, if there is a problem during processing, the message is passed to theself.error_channel
.Parameters: event_message (GEventMessage) – message requiring additional information.
Event Consumer¶
Client module to consume Google Cloud Pub/Sub messages and create an
event message to be passed around the gordon
core system and its
plugins. Once an event message is created, it will be placed into the
success_channel
to be further handled by the gordon
core system.
While being processed by the other plugin(s), the consumer will
continuously extend the message’s ack
deadline. When an event
message is done (either successfully published or met with errors) the
consumer will then ack
the message in Pub/Sub to signify that the
work is complete.
Attention
The event consumer client is an internal module for the core gordon logic. No other use cases are expected.
-
class
gordon_gcp.
GPSEventConsumer
(config, success_channel, error_channel, metrics, subscriber, flow_control, validator, parser, loop, **kwargs)[source]¶ Consume messages from Google Cloud Pub/Sub.
Pub/Sub messages are continually consumed via google-cloud-python’s pubsub module using gRPC. Every consumed message will create an asyncio task that handles the message schema validation, creation of a
GEventMessage
instance (event_msg), and the forwarding on for further processing via theself.success_channel
. The pubsub module handles the message ack deadline extension behind the scenes. Once the event_msg is done processing, gordon’s core routing system will submit it back to this consumer to be ack’ed via the handle_method method used for pubsub cleanup.Parameters: - config (dict) – configuration relevant to Cloud Pub/Sub.
- success_channel (asyncio.Queue) – a sink for successfully
processed
interfaces.IEventMessages
. - error_channel (asyncio.Queue) – a sink for
interfaces.IEventMessages
that were not processed due to problems.
GDNS Publisher¶
Client module to publish DNS records from an event message. Once an event
message is done (either successfully published, or met with errors along the
way), it will placed into the appropriate channel, either the
success_channel` or error_channel
to be further handled by the gordon
core system.
Attention
The publisher client is an internal module for the core gordon logic. No other use cases are expected.
-
class
gordon_gcp.
GDNSPublisher
(config, metrics, dns_client, **kwargs)[source]¶ Publish records to Google Cloud DNS.
Parameters: - config (dict) – Configuration relevant to Cloud DNS.
- success_channel (asyncio.Queue) – A sink for successfully
processed
interfaces.IEventMessages
. - error_channel (asyncio.Queue) – A sink for
interfaces.IEventMessages
that were not processed due to problems. - dns_client (gdns.GDNSClient) – A Google DNS HTTP connection class.
Gordon Janitor¶
Reconciler¶
Module to compare desired record sets produced from a Resource Authority
(i.e. GCEInstanceAuthority
for Google Compute Engine) and actual
record sets from Google Cloud DNS, then publish corrective messages to
the internal changes_channel
if there are differences.
This client makes use of the asynchronous DNS client as defined in
GDNSClient
, and therefore must use service account/JWT
authentication (for now).
See Janitor Configuration for the required Google DNS configuration.
Attention
This reconciler client is an internal module for the core janitor logic. No other use cases are expected.
To use:
import asyncio
import gordon_gcp
config = {
'keyfile': '/path/to/keyfile.json',
'project': 'a-dns-project'
}
rrset_chnl = asyncio.Queue()
changes_chnl = asyncio.Queue()
reconciler = gordon_gcp.GDNSReconciler(
config, rrset_chnl, changes_chnl)
loop = asyncio.get_event_loop()
try:
loop.run_until_complete(reconciler.start())
finally:
loop.close()
The keyfile is optional. If not provided the default service account will be used.
-
class
gordon_gcp.
GDNSReconciler
(config, metrics, dns_client, rrset_channel=None, changes_channel=None, **kw)[source]¶ Validate current records in DNS against desired source of truth.
GDNSReconciler
will create a change message for the configured publisher client plugin to consume if there is a discrepancy between records in Google Cloud DNS and the desired state.Once validation is done, the Reconciler will emit a
None
message to thechanges_channel
queue, signalling a Publisher client (e.g.GPubsubPublisher
) to publish the message to a pub/sub to which Gordon subscribes.Parameters: - config (dict) – Google Cloud DNS-related configuration.
- metrics (obj) –
IMetricRelay
implementation. - dns_client (GDNSClient) – Client to interact with Google Cloud DNS API.
- rrset_channel (asyncio.Queue) – Queue from which to consume record set messages to validate.
- changes_channel (asyncio.Queue) – Queue to publish message to make corrections to Cloud DNS.
-
cleanup
()[source]¶ Clean up & notify
changes_channel
of no more messages.This method collects all tasks that this particular class initiated, and will cancel them if they don’t complete within the configured timeout period.
Once all tasks are done,
None
is added to thechanges_channel
to signify that it has no more work to process. Then the HTTP session attached to thedns_client
is properly closed.
-
static
create_rrset_set
(zone, rrsets, source=None)[source]¶ Create a set of ResourceRecordSets excluding SOA and zone’s NS.
Parameters: Returns: set of
ResourceRecordSet
-
publish_change_messages
(desired_rrsets, action='additions')[source]¶ Publish change messages to the
changes_channel
.NOTE: Only ‘additions’ are currently supported. ‘deletions’ may be supported in the future.
Parameters:
-
run
()[source]¶ Publish necessary DNS changes to the
changes_channel
.Consumes zone/rrset-list messages from
rrset_channel
, compares them to the current records, and publishes the changes. OnceNone
is received from the channel, emits a finalNone
message to thechanges_channel
.
-
validate_rrsets_by_zone
(zone, rrsets)[source]¶ Given a zone, validate current versus desired rrsets.
Returns lists of missing rrsets (in desired but not in current) and extra rrsets (in current but not in desired). Extra rrsets that are the result of updates in the desired list will not be returned, and root SOA/NS comparisons are skipped.
Parameters: Returns: The missing and extra rrset sets.
Return type:
GPubSub Publisher¶
Client module to publish any required DNS changes initiated from
GDNSReconciler
to Google Cloud Pub/Sub. The consumer of these messages is
the Gordon service.
This client wraps around google-cloud-pubsub using grpc rather than
inheriting from AIOConnection
.
Attention
This publisher client is an internal module for the core janitor logic. No other use cases are expected.
To use:
import asyncio
import gordon_gcp
config = {
'keyfile': '/path/to/keyfile.json',
'project': 'a-dns-project',
'topic': 'a-topic',
}
changes_channel = asyncio.Queue()
publisher = gordon_gcp.get_gpubsub_publisher(
config, changes_channel)
loop = asyncio.get_event_loop()
try:
loop.run_until_complete(publisher.start())
finally:
loop.close()
The keyfile is optional. If not provided the default service account will be used.
-
class
gordon_gcp.
GPubsubPublisher
(config, metrics, publisher, changes_channel=None, **kw)[source]¶ Client to publish change messages to Google Pub/Sub.
Parameters: - config (dict) – Google Cloud Pub/Sub-related configuration, ex. ‘projects/test-example/topics/a-topic’.
- publisher (google.cloud.pubsub_v1.publisher.client.Client) – client to interface with Google Pub/Sub API.
- metrics (obj) –
IMetricRelay
implementation. - changes_channel (asyncio.Queue) – queue to publish message to make corrections to Cloud DNS.
-
cleanup
()[source]¶ Clean up outstanding tasks and emit final logs + metrics.
This method collects all tasks that this particular class initiated, and will cancel them if they don’t complete within the configured timeout period.
Authority¶
A GCEAuthority retrieves a list of all instances in all projects that it has access to, and which belong to the configured zone. For every project, it will create a message containing domain record information and put it into the rrset channel. Projects can be filtered by ‘project name’. Instances can be filtered by tags and metadata.
To use:
import asyncio
from gordon_gcp.plugins import janitor
async def run():
rrset_channel = asyncio.queue()
authority = janitor.get_authority(config, rrset_channel)
await authority.start()
msg = await rrset_channel.get()
print(msg)
loop = asyncio.get_event_loop()
loop.run_until_complete(run())
# prints: {'zone': 'example.com', 'resourceRecords': [...]}
-
class
gordon_gcp.
GCEAuthority
(config, metrics, crm_client, gce_client, rrset_channel=None, **kwargs)[source]¶ Gather instance data from GCE.
Parameters: - config (dict) – plugin-specific configuration.
- metrics (obj) –
IMetricRelay
implementation. - crm_client (GCRMClient) – client used to fetch GCE projects.
- gce_client (GCEClient) – client used to fetch instances for a project.
- rrset_channel (asyncio.Queue) – channel to send resource record messages to.