You are currently viewing How to configure custom logs in OCI for any type of workload

How to configure custom logs in OCI for any type of workload

This post is part of the OCI Logging – Complete Hands-on Series. Make sure to check out the other posts as well.

Definition

  • Custom logs: Logs that contain diagnostic information from custom applications, other cloud providers, or an on-premise environment. Custom logs can be ingested through the API, or by configuring the Unified Monitoring Agent. You can configure an OCI compute instance/resource to directly upload Custom Logs through the Unified Monitoring Agent. Custom logs are supported in both a virtual machine and bare metal scenario.

Here we talk about any types of logs from Compute VMs or DBCS instances running on OCI to VMs running on-premises or in any other cloud. The possibilities are endless – all you need is to have the OCI Unified Monitoring Agent running on the machine from where you want to get the logs from.

Setup

Ingesting Custom Logs into OCI is very simple, using one of the following ways:

  • By configuring the Unified Monitoring Agent (the agent should be already installed and configured for all OCI images). See Installing the Agent for instructions for any other scenario.
  • By using PutLogs to ingest custom logs directly. See the Logging Ingestion API and REST APIs for more information. Also, see Using the API for an example log entry payload that can be used with PutLogs.

We’ll look at how to ingest logs from an OCI Compute Instance using the OCI Unified Monitoring Agent.

Use Case Description

Short description of the use case: I have one compartment where I have 2 Compute VM Instances with the OCI Unified Monitoring Agent preinstalled and I will extract the system logs from /var/log/messages from both VMs into OCI Logging.

Prerequisites

OCI Agent Setup

The Agent is automatically installed on all OCI instances using the supported OS:

  • Oracle Linux 7, Oracle Linux 8
  • CentOS 7, CentOS 8
  • Windows Server 2012 R2, Windows Server 2016, Windows Server 2019
  • Ubuntu 16.04, Ubuntu 18.04, Ubuntu 20.04

    You just need to make sure the Custom Log Monitoring plugin is enabled and running on your instance.

    To do this, navigate to Compute -> Instances -> Your Instance -> Oracle Cloud Agent:

    If you plan to ingest logs from somewhere else or the Agent is not installed for some reason, please check the Agent Installation steps.

    Dynamic Group

    Before creating the Custom Log, we need to define a Host Group which will point to the VMs from where we want to get logs from. The Host Groups can be defined either by using Dynamic Groups or User Groups

    In this example, we’ll use Dynamic Groups. With Dynamic Groups, we can point at either a singular VM:

    instance.id = <instance OCID>

    or at multiple VMs:

    any|all {<instance.id= <instanceOCID>>,<instance.id= <instanceOCID>>,...}

    or we can point at a compartment where the instance(s) reside:

    instance.compartment.id = <compartment OCID>

    The latter, using the compartment OCID, is the most flexible one as we don’t need to reconfigure the logging each time there’s a new instance created. This is how I’ll be defining my Dynamic Group as well.

    So, let’s navigate to the Main Menu -> Identity & Security -> Dynamic Groups and create a new Dynamic Group using the compartment OCID:

    Check the documentation for more info on how to use the Dynamic Groups.

    Create the Custom Log

    Now that we have our Host Group defined as a Dynamic Group, we can navigate to the Main Menu -> Observability & Management -> Logs (under the Logging Category) and Click on the Create Custom Log at the top of your screen:

    In the first screen, give the log a name, place it in a compartment and in a Log Group.

    In this screen, we’ll define the Agent Configuration for our log: Give the Agent Configuration a name, a description and select the Host Group created earlier as Dynamic Group

    Also, click on Create to automatically create the necesary policy to allow the instances in the Dynamic Group to use the Logging Service

    Next we define the Log Input. We’ll select Log Path input type and we’ll give this log input a name, specify the file path(s) (you can also use * wildcard in your file path definition) and click on Create Custom Log.

    You can easily add multiple File Paths in one Log Input, or you can create multiple Log Inputs by clicking the + Another log input button.

    You can select Windows Event Logs as well in case of Windows operating system

     

    Go to Advanced Parser Options. This allows you to specify how to parse the log.

    We’ll select SYSLOG for this example.

    You can also modify the Additional Options to your liking. In my case, I’ll just leave everything with the default values.

    After the custom log configuration was created – it will take a few minutes before any logs are ingested into OCI. Also, make sure that you have new messages that are being written in your logs.

    View the logs

    You can view the logs either from the page of the newly created log (Main Menu -> Observability & Management -> Logs and select your log) or from the Search menu in the Logging category (Main Menu -> Observability & Management -> Search and filter your search accordingly) or even directly from the Compute Instance Details Page (Main Menu -> Compute -> Instances -> Your Compute Instance -> Custom Logs menu under Resources).

    Let’s just go to the newly created log and see the logs from there now:

    You can expand a log entry using the little arrow on the right-hand side and you can see the whole content of your log plus some additional paramters added by OCI.

    You can check out this post on How to search logs efficiently in OCI.

    Congratulations – you now have your Custom Logs ingested in OCI!

    Image Rights

    Ionut Adrian Vladu

    I enjoy building python scripts for…everything! I am a Cloud enthusiast and I like to keep up with technology. When I'm not behind a computer, I like taking photos -- Visit My 500px profile -- or sit back and enjoy Formula 1 race weekends. Currently, working as a Tech Cloud Specialist @ Oracle
    Subscribe
    Notify of
    guest

    0 Comments
    Inline Feedbacks
    View all comments