In this short post, I'm going to share how you can get more out of your logging on Google Cloud Platform with Cloud Functions. Although I'm going to use Node.js runtime, the language is irrelevant for this tutorial.
My use case originated from SendGrid Activity Feed, which is a list of all events that occurred with emails I sent (processed, delivered, deferred, blocked, etc.). The issue is the feed is limited to 500 items or 30 days, and I needed more visibility into what's happening. The great thing is that SendGrid allows setting the Event Webhook, which I can use to collect all events on my own. I wanted a quick to implement, easy to maintain, and cheap solution and function that's just logging payloads I can then query seems to be a great fit.
Structured Logging
First of all, what's a structured log? Google Cloud's operations suite (formerly Stackdriver) that underlies Cloud Logging infrastructure makes a distinction between plain text and structured JSON logs. Structured logs enable, among others, much more fine-grained queries but also can be feed into BigQuery or Cloud Pub/Sub.
I didn't want to use Google Cloud Logging client to keep the function as small as possible, and I don't need to set custom metadata for my logs. The side product is that it makes this post more or less language-agnostic.
exports.webhook = (req, res) => {
req.body.forEach((event) => {
console.log(JSON.stringify({ event })); });
res.status(200).send({ status: "ok" });
};The key takeaway is that you have to log valid JSON, so in JavaScript functions, you have to stringify your log first. Act accordingly, whatever programming language you use.
Query Logs
In Google Cloud Dashboard under Operations, select Logging and go to Logs Viewer. Switch to advanced mode for query input and type your query. You can read more about advanced queries in Logging docs. In my case, where I want to get a list of all delivered emails, I can just type
resource.type="cloud_function"
resource.labels.function_name="events-hook"
resource.labels.region="us-central1"
jsonPayload.fields.event.Kind.StructValue.fields.event.Kind.StringValue="delivered"Take some time to explore the Logs Viewer. If it makes sense for you can use the query to create a metric or sink to export data to Cloud Storage, BugQuery, or Pub/Sub. In case you plan to revisit this query more often, you can save it in a library. By selecting a field in the result, you can add it or hide it from the summary line.
Sensitive Information
Only trusted parties should be able to access the logs of your services. Still, if you are obligated to anonymize data or require an additional protection layer, you can filter out the sensitive information from your log. Do not forget about it. To obfuscate email addresses from my logs, I could use a simple anonymization function like this one
function anonymizeEmail(str) {
return str.replace(
/([a-z0-9._%+-]+)(@[a-z0-9._%+-]+)/gi,
(_full, user, rest) => `${user.replace(/./g, "*")}${rest}`,
);
}Summary
If I think about it, this is probably the least amount of code I've written that provides me with business value. When you need to take it a step further, check out Google Cloud Logging Client for your language of choice. I hope you found this post helpful!
Photo by Brendan Church on Unsplash.