Improved control + security with real-time logging
VP Strategic Initiatives, Fastly
We know that our customers value visibility and control — you need actionable insight into what’s going on across your digital services, and the flexibility to make changes when necessary. Real-time logging lets you see what’s happening with your traffic, empowering you to make decisions and changes on the fly. With Fastly logging, you can instantly stream your log files to a provider of your choosing (see the full list here), and dig into any aspect of a request or response to diagnose problems and understand how your customers are engaging with your app. And, because we believe your log data belongs to you, we don’t store logs.
Leading organizations use logging in a variety of innovative ways. The Guardian use log streaming as an early warning system to detect issues after changes are deployed to their site; Foursquare decides what content is cached as well as what data is streamed using logs; Upworthy uses Fastly stats alongside other cloud providers to give their team a holistic and real-time account of site health; the tech team at Target starts their day with custom-built dashboards that allow them to collect and develop knowledge on what was happening at the edge of their network, and integrating that into deeper knowledge at other parts in their stack.
Our edge cloud platform handles approximately 4 million log lines per second, using close to 5 Gbps of bandwidth, offering you a comprehensive view of your traffic: you can use log data to retrospectively review system behavior, as well as react in real time during an event (e.g., signaling potential origin overload if the response time for a cache miss is more than five seconds). You can also programmatically modify your Fastly logs using scripts that call our API whenever your log data meets predefined conditions (e.g., enforcing rate limiting or blacklists by adding IPs to your versionless edge ACLs based on request information)
Big Cartel proactively monitors traffic patterns and instantly updates their configurations to block malicious traffic at the edge
In this post, we’ll share our latest logging updates — which allow for improved formatting and control, new and improved logging endpoint integrations, and enhanced security — and how to get started.
Compatibility updates
We’ve added support for several new logging endpoints and improved our existing integrations in order to make it easier for you to stream logs to your favorite endpoint.
One-click integration with Logentries
We’ve partnered with Logentries to make it as easy as possible to send logs from Fastly to their log analysis platform. You can now press one button in our control panel to create a trial Logentries account and configure a logging object to send to it. You will not be charged for this trial account unless you choose to keep it after the 30-day Logentries trial is up, and you can delete the automatically generated log object at any time.
Google BigQuery support
You can now stream logs directly into Google's BigQuery big data platform, which allows you to perform sophisticated analysis on large datasets, correlate data with other sources writing to BigQuery, and do data visualization with tools such as Looker and Tableau. (You can check out logging to BigQuery with this Fiddle.)
Example troubleshooting dashboard in Looker with Fastly data
Sumo Logic dashboard
With the new Sumo Logic App for Fastly, you can access a set of preconfigured dashboards that will help you to quickly analyze and correlate relevant data. You can further refine your dashboards as you see fit, and create notification alerts when thresholds are reached.
Premade Sumo Logic dashboard for origin performance
Scalyr support
You can now send logs to Scalyr, a popular logging-as-a-service provider frequently requested by customers.
Improved formatting and control
Our latest updates offer you more control over the way your logs are delivered:
We’ve upgraded our formatting system for full compatibility with Apache Log directives for better interoperability with external systems. Any third-party documents or blog posts about Apache Logging will now work with Fastly. In addition, the new parser makes it much easier to have structured logging formats, such as CSV and JSON, and to use any VCL expression in your log lines without having to use Custom VCL.
Now you can modify the format of the message sent out, making interaction with third parties easier. Previously we sent log messages in standard syslog format which includes a prefix (as defined in RFC 3164), as seen in the example below:
<134>2016-07-04T22:37:26Z cache-sjc3128 LogTest[62959]: <your log message>We now allow you to choose one of several formats:
Classic is the default prefix format. A standard syslog prefix as defined by RFC 3164.
Logplex is a Heroku-style length prefixed syslog format.
Blank means no prefix — just your log message, which means it's now possible to generate JSON and CSV files.
We’ve also provided the ability to customize the timestamp that’s included when we write log files to endpoints like S3, GCS, and FTP. By default, we add the time the log is created in ISO 8601 format, which looks like
2017-02-10T19:55:42.000.However, sometimes you need that timestamp to be formatted differently — for example, services such as Amazon’s ElasticMapReduce may have problems with files that have colons in the file name. Now you can set the format of the timestamp using standard strftime format. A handy online tool allows you to experiment with the format here, which will help you figure out the best format for your needs.
Our logs can surface nearly anything that’s available in the request or response header, including client IP, timestamp, request type, request URL, and HTTP status code. We’ve written extensions to VCL that provide:
Time and date variables: includes data that can provide additional flexibility when dealing with dates and times.
Size-related variables: add request and response size to your logs, including a breakdown of header and body size.
Geolocation-related variables: expose a number of geographic variables based on geolocation, including continent, country, and city associated with the client IP. Geographical region and POP location (i.e., city) of the data center that requests come through is also available.
Do it yourself
With the ability to more tightly control every aspect of how your log line is created customers should now find it easier to connect to Logging services that Fastly doesn't directly support.
For example, customers who want to connect to LogDNA can use the generic Syslog connector and configure the hostname and TLS certificate themselves. We've documented the procedure in our Streaming Logs Guide here.
As a more complicated example, if Fastly didn't support Loggly then it would be similarly possible to send logs there by:
Setting the hostname to logs-01.loggly.com
Setting the port to 6514
Setting TLS to yes
Setting the message type to blank
Getting your token from Loggly
Setting the format to
<134>1 %{"%Y-%m-%dT%T%z"}t %{server.datacenter}V <log name> - - [<token>@41058 tag="fastly" tag="other tag"] <rest of format>
That last bit requires some explanation. What this is doing is creating a syslog message using the format described in RFC 5424 which is the format that Loggly expects according to their documentation. We insert our token in the "Structured Data Element" part of the message along with their PEN (Private Enterprise Number), 41058, along with any tags we might want. The rest of the format is just the regular logging message as normal.
We have had customers do the same thing when wanting to send logs to Logz.io and to Datadog's new Logging product.
In the future we'll add those as first class Logging objects for ease of use but, in the meantime, this flexibility is part of Fastly's philosophy that we should strive to allow customers to integrate with whatever systems they'd like, whether Fastly natively supports it or not. In the future we also want to make it easier for customers to roll-their-own integrations with HTTP based logging and analytics systems such as New Relic Insights as well.
Enhanced security
For additional security, we’ve added the ability for customers to turn on server-side encryption when writing to Amazon S3. When enabled, any log files we send are encrypted as they’re written to disk, protecting your data.
As this is an Amazon-only feature, we’ve also added support for a similar feature which allows you to give us a PGP (Pretty Good Privacy, the defacto standard for encryption) public key so that we can encrypt any log files we write for you. You can then decrypt the files at your leisure, using your own private key. This works for any logging endpoint that writes files, which includes Amazon S3, Google Cloud Storage (GCS), and File Transfer Protocol (FTP).
Going forward
We hope these updates will help you get the most out of your logs and better enable you to leverage your data in real time — ultimately giving you actionable insights into requests and responses. For more info on Fastly real-time logging (plus how to get started), visit our guide, and check out this in-depth workshop on logging at the edge from Altitude, our customer summit in San Francisco. As always, if you have requests for features or logging providers you'd like to see, don’t hesitate to let us know.
Happy logging!
