In this article, we will see how we can manage logs generated by APIGEE API gateway with Elastic Search, Logstash and Kibana. The below diagram highlights how log data flows through ELK stack for data visualization and monitoring.
As part of the API gateway, we will use MessageLogging policy to capture API proxy logs at proxy flow, target flow and post client flow based on the requirement. As of today, MessageLogging policy supports Syslog and file-based logging(available for only on-premise installations). Below is MessageLogging policy configuration to send logs to the syslog server.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <?xml version="1.0" encoding="UTF-8" standalone="yes"?> | |
| <MessageLogging async="false" continueOnError="false" enabled="true" name="TestEnv-Message-Logging"> | |
| <DisplayName>TestEnv Message Logging</DisplayName> | |
| <Syslog> | |
| <Message>{messageId}|{apiproxy.name}|{response.status.code}|{target.received.start.timestamp}|{target.received.end.timestamp}|{environment.name}|{client.received.start.timestamp}|{client.sent.end.timestamp}</Message> | |
| <Host>127.0.0.1</Host> | |
| <Port>514</Port> | |
| </Syslog> | |
| </MessageLogging> |
Now we will see logstash configuration to ingest data from syslog to logstash. Here I am not giving steps to setup ELK as that is out of scope.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| input { | |
| file { | |
| path => "/Users/sjana/Documents/POC/ELK/APIGEE_LOG.log" | |
| start_position => "beginning" | |
| sincedb_path => "/dev/null" | |
| } | |
| } | |
| filter { | |
| csv { | |
| separator => "|" | |
| columns => ["MessageId","API_PROXY_NAME","RESPONSE_STATUS_CODE","TARGET_RECEIVED_START_TIME","TARGET_RECEIVED_END_TIME","ENVIRONMENT_NAME","CLIENT_RECEIVED_START_TIME","CLEINT_SENT_END_TIME"] | |
| remove_field => "message" | |
| } | |
| } | |
| output { | |
| elasticsearch { | |
| hosts => "http://localhost:9200" | |
| index => "apiproxies-log" | |
| } | |
| stdout {} | |
| } |
Once Syslog data is ingested successfully to Logstash we can configure visualization based on available fields in the index. Below is sample visualization created for API proxy response status codes.
In coming articles we will discuss another topic. Till then, Spread love for APIs!!!
Siva Janapati is an Architect with experience in building Cloud Native Microservices architectures, Reactive Systems, Large scale distributed systems, and Serverless Systems. Siva has hands-on in architecture, design, and implementation of scalable systems using Cloud, Java, Go lang, Apache Kafka, Apache Solr, Spring, Spring Boot, Lightbend reactive tech stack, APIGEE edge & on-premise and other open-source, proprietary technologies. Expertise working with and building RESTful, GraphQL APIs. He has successfully delivered multiple applications in retail, telco, and financial services domains. He manages the GitHub(https://github.com/2013techsmarts) where he put the source code of his work related to his blog posts.
Smart Techie 
Leave a Reply