Category: Virtualisation (Page 3 of 8)

vRealize Log Insight + PKS Integration

February 2, 2019 / David / 3 Comments

Introduction

In this blog post, we take a look into the integration between PKS and vRealize Log Insight and how this integration benefits the enterprise. As a bit of a recap:

PKS – PKS is a purpose-built enterprise level container solution leveraging the capabilities of Kubernetes, BOSH, VMware NSX-T, Harbour and more to deliver a highly available, highly flexible container runtime that operates on a number of cloud platforms, both private and public, including vSphere, AWS, Azure and GCP.

VMware also released VMware Cloud PKS, a fully managed service that combines the technical capabilities of AWS, PKS and Kuberntes which can be consumed in a similar fashion to other cloud services.

vRealize Log Insight – vRealize Log Insight is a log management system that’s designed to operate within heterogeneous environments, however, it’s much more than a simple aggregator of logging information. vRealize Log Insight has analytical and trend-identification capabilities which allow operators to gain invaluable insight into the state, health, and events which are transpiring in the environment. vRealize Log Insight works across physical, virtual and cloud environments.

Containers and Coexistence with VM’s

VM’s have existed for a long time now. Consequently, there are very mature, battle-hardened tools and software which can be used to monitor a plethora of operating systems, software, components and more. Containers, on the other hand, are relatively new in the enterprise. Although there is an overlap, there are significant differences in the way we monitor and collect logs from VM’s and containers. How can this be addressed?

There are a number of ways to monitor a container based environment. Prometheus and Wavefront come to mind, but for environments that already leverage vRealize Log Insight, we can integrate PKS with it to facilitate a single plane of glass view of logging information from VM’s, their underlying infrastructure as well as containers and their underlying infrastructure.

What can we expect PKS to send to Log Insight

At a high level, the Integration between PKS and vRLI will facilitate the propagation of the following logs:

BOSH jobs
Core Kubernetes processes & nodes
Core BOSH processes
Kubernetes event logs
Individual Pod stdout and stderr

I’ve highlighted the last one as I can see real value in this. Imagine centralising all stdout and stderr from pods in combination with the analytics and trend identification capabilities from vRLI? Pretty interesting. Of course, we’re not that interested in what individual pods are logging, but if we have an example where some new code has been pushed out and 10’s / 100’s or 1000’s of pods start logging errors, we can identify, categorise and analyse these pretty easily with vRLI.

PKS and vRealize Log Insight in action

Talk is cheap, so let’s crack on.

Log into Ops Manager and select the PKS tile

Select “Logging” from the left and select “yes” under vRLI integration:

Enter the host and SSL settings where applicable in your environment:

Apply the changes:

if you keep an eye on the logs, references for the vRLI configuration will be shown:

– fluentd_vrli_ca_cert: “<redacted>”
– fluentd_vrli_host: “<redacted>”
+ fluentd_vrli_host: “<redacted>”
– fluentd_vrli_rate_limit_msec: “<redacted>”
+ fluentd_vrli_rate_limit_msec: “<redacted>”
– fluentd_vrli_skip_cert_verify: “<redacted>”
+ fluentd_vrli_skip_cert_verify: “<redacted>”
– fluentd_vrli_use_ssl: “<redacted>” + fluentd_vrli_use_ssl: “<redacted>”

Next, deploy a cluster in PKS:

After which, the following “hosts” can be observed, which in essence, is a reflection of the services within our Kubernetes cluster:

I also create a individual pod, named nginx-sleep. Below are the logs that were ingested for this event:

To validate the stdout capturing, create a cluster that writes to stdout:

And check the logs from the pod:

And also from Log Insight:

Conclusion

vRealize Log Insight provides a compelling platform for log ingestion, and it’s flexibility to ingest, analyse and interpret logs from physical, virtual and container based solutions makes it an extremely versatile tool in any admins repertoire.

vSphere and Containers part 1 – VIC (VMware Integrated Containers)

March 28, 2018 / David / 0 Comments

In this multi-part series, we evaluate the options available to vSphere users/customers wishing to deploy a native container service into an existing vSphere environment.

Part 1 – VIC (VMware Integrated Containers).

Part 2 – PKS (Pivotal Container Service).

Why should we care about containers?

Containers change the way we fundamentally look at application deployment and development. There was a huge shift in the way we managed platforms when server virtualisation came around – all of a sudden we had greater levels of flexibility, elasticity and redundancy compared to physical implementations. Consequently, the way in which applications were developed and deployed changed. And here we are again, with the next step of innovation using technology that is making rifts in the industry, changing the way consume resources.

What is VIC?

VIC (or vSphere Integrated Containers) is a native extension to the vSphere platform that facilitates container technology, because of this tight integration we’re able to perform actions and activities using the vSphere client and integrate it with auxiliary services. VIC is developed in such a way so it presents a Docker Compatible API endpoint. Therefore Ops/Dev staff already familiar with Docker can leverage VIC using the same tools/commands that they’re already familiar with.

VIC is a culmination of three technologies:

The containers engine is the core runtime technology that facilitates containerised applications in a vSphere environment. As previously mentioned, this engine presents a Docker-compatible API for consumption. Tight integration between this and vSphere enables vSphere admins to manage container and VM workloads in a consistent way.

Harbour is an enterprise-level facilitator of Docker-based image retrieval and distribution. It’s considered an extension of the open source Docker Distribution by adding features and constructs that are beneficial to the enterprise including but not limited to : LDAP support, Role-based access control, GUI control and much more.

Admiral is a scalable and lightweight container management platform for managing containers and associated applications. Primary responsibilities are mainly around automated deployment and lifecycle management of containers.

How VIC works

The management plane of VIC is facilitated by a OVA appliance, rather than going through the installation steps here, I will simply point to the direction of the (excellent) documentation located at https://vmware.github.io/vic-product/#documentation. At the core though, we have the following constructs:

VIC Appliance – Management plane.
Virtual Container Hosts – Infrastructure resource with a docker endpoint.
Registry – Location for Docker-compatible images.

Which, from a logical view looks like this:

Key observations are:

The VCH (Virtual Container Host) isn’t a Virtual machine, it’s actually a resource pool. Therefore, I think the best way to describe a VCH is a logical representation of a pool of resources, including clustering, scheduling, vMotion, HA, and other features.
When a VCH is created, a VM is created that facilitates the Docker-compatible API endpoint.

Advantages of VIC

So why would any of us consider VIC instead of, for example, standard Docker hosts? Here are a few points I’ve come across:

Native integration into vSphere.
Administrators can secure and manage VM and Container resources in the same way.
Easy integration into other VMware products.
1. NSX.
2. VSAN.
3. vRealize Network Insight.
4. vRealize Orchestrator.
5. vRealize Automation.
Eases adoption.
Eases security.
Eases management.

Conclusion

VIC helps bridge the gap between Developers and Administrators when it comes to the world of containers. I would say VIC is still in its infancy in terms of development, but it’s being backed by a great team and I think it’s going to make a compelling option for vSphere customers/users looking to embrace the container world, whilst maintaining a predictable, consistent security and management model.

vRealize Log insight – Frequently Overlooked Centralised Log Management

January 15, 2018 / David / 0 Comments

Log analysis has always been a standardised practice for activities such as root cause analysis or advanced troubleshooting. However, ingesting and analysing these logs from different devices, types, locations and formats can be a challenge. In this post, we have a look at vRealize Log Insight and what it can deliver.

What is it?

vRealize Log Insight is a product in the vRealize suite specifically designed for heterogeneous and scalable log management across physical, virtual and cloud-based environments. It is designed to be agnostic across what it can ingest logs from and is therefore valid candidate in a lot of deployments.

Additionally, any customer with a vCenter Server Standard or above license is entitled to a free 25 OSI pack. OSI is known as “Operating System Instance” and is broadly defined as a managed entity which is capable of generating logs. For example, a 25 OSI pack license can be used to cover a vCenter server, a number of ESXi hosts and other devices covered either natively or via VMware Content Packs (with the exception of Custom and 3^rd party content packs – standalone vRealize Log Insight is required for this feature).

Current Challenges

Modern datacenters and cloud environments are rarely consumed by homogeneous solutions. Customers use a number of different technologies from different vendors and operating systems. With this comes a number of challenges:

The inconsistent format of log types – vCenter/ESXi uses syslog for logging, Windows has a bespoke method, applications may simply write data to a file in a specific format. This can require a number of tools/skills to read, interpret and action from this data.
Silos of information – The decentralised nature of dispersed logging causes this information to be siloed in different areas. This can have an impact on resolution times for incidents and accuracy of root cause analysis.
Manual analysis – Simply logging information can be helpful, but the reason why this is required is to perform the analysis. In some environments, this is a manual process performed by a systems administrator.
Not scalable – As environments grow larger and more complex having silos of differentiating logging types and formats becomes unwieldy to manage.
Cost – Man hours used to perform manual analysis can be costly.
No Correlation – Siloed logs doesn’t cater for any correlation of events/activities across an environment. This can greatly impede efforts in performing activities such as root cause analysis.

Addressing Challenges With vRealize Log Insight

Below are examples of how vRealize Log Insight can address the aforementioned challenges.

Create structure from unstructured data – Collected data is automatically analysed and structured for ease of reporting.
Centralised logging – vRealize Log Insight centrally collates logs from a number of sources which can then be accessed through a single management interface.
Automatic analysis – Logs are collected in near real-time and alerts can be configured to inform users of potential issues and unexpected events.
Scalable – Advanced licenses of vRealize Log insight include additional features such as Clustering, High Availability, Event Forwarding and Archiving to facilitate a highly scalable, centralised log management solution. vRealize Log Insight is also designed to analyse massive amounts of log data.
Cost – Automatic analysis of logs and alerting can assist with reducing man-hours spent manually analysing logs, freeing up IT staff to perform other tasks.
Log Correlation – Because logs are centralised and structured events across multiple devices/services can be correlated to identify trends and patterns.

Extensibility

vRealize Log Insight’s capabilities can be extended by the use of content packs. Content packs are available from the VMware marketplace (https://marketplace.vmware.com/vsx/?contentType=2)

Content packs are published either by VMware directly or from vendors to support their own devices/solutions. Examples include:

Apache Web Service
Brocade Devices
Cisco Devices
Dell | EMC Devices
F5 Devices
Juniper Devices
Microsoft Active Directory
Nimble Devices
VMware SRM

Closing Thoughts

It’s surprising how underused vRealize Log Insight is considering it comes bundled in as part of any valid vSphere Standard or above license. The modular design of the solution allowing third-party content packs adds a massive degree of flexibility which is not common amongst other centralised logging tools.