Saturday, June 13, 2020

Pantheios.Ruby Tutorial, part 2: First Program

This is the second part in a series of tutorial posts illustrating how to use Pantheios.Ruby. The first part is here.

Project Setup

In this part we're going to do some basic project setup. Expert Rubyists may wish to skip to the next section.

First step is to create a git project. Create a new directory and do a Git init:

$ git init

I'm going to use Bundler, so next do a Bundle init:

$ bundle init

I happen to know that we'll be using recls.Ruby as well as Pantheios.Ruby, so let's edit the generated bundle file to look something like the following:

  # frozen_string_literal: true
  
  source "https://rubygems.org"
  
  git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }
  
  gem "pantheios-ruby"  
  gem "recls-ruby"  

Then just bundle it:

$ bundle

All being well, add Gemfile and the newly generated Gemfile.lock into git and commit.

First Program

Ok, now on to the first program. First create the script searcher.rb and require and include both libraries, as follows:

  1 #! /usr/bin/env ruby
  2 
  3 require "pantheios"
  4 require "recls"
  5 
  6 include ::Pantheios
  7 

Give it a run to verify Bundler did its job ok.

$ ruby searcher.rb

Go ahead and commit that, then we'll make it a little interesting by doing a search and logging some output.

  1 #! /usr/bin/env ruby
  2 
  3 require "pantheios"
  4 require "recls"
  5 
  6 include ::Pantheios
  7 
  8 Recls.file_rsearch(nil, nil).each do |fe|
  9 
 10     log(:informational, "found '#{fe.search_relative_path}'")
 11 end 
 12    

This uses the Recls module function file_rsearch() to conduct a recursive search for files, returning an Enumerable object on which we call each passing the block that uses Pantheios.Ruby's log() method to record each item found. On my machine it produces the following output

[searcher, 70123163703220, 2020-06-14 09:31:35.142106, Informational]: found 'searcher.rb'
[searcher, 70123163703220, 2020-06-14 09:31:35.142559, Informational]: found 'Gemfile'
[searcher, 70123163703220, 2020-06-14 09:31:35.143248, Informational]: found 'Gemfile.lock'

In the next instalment we'll examine the aspects of this program and its output and make some customisations to what gets logged and in what form.


Friday, June 12, 2020

Pantheios.Ruby Tutorial, part 1: Basic Script

So, this'll be the first of a set of tutorials explaining how to use Pantheios.Ruby.

Install and Verify


The first thing to do is install it, which is done as simply as:

$ gem install pantheios-ruby

(You may need/prefer to download the latest gem from rubygems and install from a local gem file. You may need the right privileges, so could require sudo.)

You can then verify the installation with the one-line:

$ ruby -e 'require "pantheios" ; puts Pantheios::VERSION;'

and should see output such as:

0.22.0

Basic Script

Next, we'll create a minimally-featured script that does some logging. Using vim - I mean, what else is there?! - create a script hello_pantheios.rb with the following contents:

  #! /usr/bin/env ruby
  
  require "pantheios"
  
  include ::Pantheios
  
  log(:notice, "Hello, World of Logs!")

and then execute as follows:

$ ruby hello_pantheios.rb

you'll see output such as:

[hello_pantheios, 70345600268220, 2020-06-13 10:09:21.867107, Notice]: Hello, World of Logs!

Coming up:

In the next post we'll create a first version of a file-system search program with which we will illustrate a whole raft of Pantheios.Ruby features, including (but not limited to):

  • using different severity levels and ascribing meaning to those levels;
  • filtering based on severity levels, including using front-ends to make decisions about which severity levels will be output;
  • choosing back-ends, i.e. where logged output goes, including:
    • logging to the console;
    • logging to the console with severity level-related colours;
    • files;
    • syslog;
    • Ruby's standard logger;
    • log4r;
    • ... and so on;
  • naming threads;
  • customising the prefix - the bit above that contains "[hello_pantheios, 70345600268220, 2020-06-13 10:09:21.867107, Notice]: ";
  • ... and many more things I can't think of right now. That's why we're going to write an actually useful program to exercise it.


Monday, June 8, 2020

Pantheios : what's coming up in 2020?

Well, folks, it's been a long while and not much activity on this blog. As you may've noticed, I've started blogging daily this last week, and aim to try and keep up at least a weekly update henceforth.

Despite the lack of activity on the blog there's been a lot of work on various Pantheios projects. Here's a summary of what's been happening, and what's planned for the rest of 2020:


  • The website will get a long-awaited update;
  • The Pantheios main (C/C++) project: 
    • will have fewer than ten more betas over the next couple of months;
    • the next beta off the rank will include:
      • the expanded stock severity levels;
      • an example of how to use the old severity levels;
      • examples of how to use custom severity levels;
      • will support VC++ up to 16.x (VS2019);
      • will support latest versions of GCC;
      • will have (auto-generated) solution and project files for all versions of VC++ from 10+ (VS2010+);
      • ... and a bunch more stuff;
    • will go into a final 1.0.1 release before the end of the year;
    • will use a different mechanism for linking to its dependency projects - TBD;
    • will get CMake builds this year;
    • will get installable packages early next year;
  • The Pantheios.Ruby project:
    • has been released, currently available via GitHub and as a gem;
    • will be documented soon;
  • The Pantheios.NET project:
    • has been updated just a couple of days ago with the new expanded stock severity levels, available via Nuget;
    • will be getting its own website - pantheios.net - including tutorials and samples;
    • will be open-sourced before the end of the year - currently only available as Nuget packages.
  • The Pantheios.COM project:

Also possible are either/both of the nascent Pantheios.Go and/or Pantheios.Python project being released.

Pantheios is a Diagnostic Logging *API* library

Just a reminder to any old/new users of Pantheios: it is a diagnostic logging API library. The API bit is crucial. While each language implementation of Pantheios - C/C++, C#, Ruby, ... - does come with a bunch of stock "back-ends" that allow some fairly useful logging, it is a primary design parameter that it should be used as a superior API over existing sophisticated logging libraries.

There'll be posts illustrating just how this is achieved for various languages over the coming weeks.

Sunday, June 7, 2020

Pantheios.NET / SynesisSoftware.Pantheios.NET.Extensions 0.12.1061 released


CHANGES:

  • added dynamic loading of components named in app.config from entry assembly directory
  • issues a contingent report when a thread fails with an uncaught exception
  • fixed defect in SynesisSoftware.Pantheios.Scopes' Pantheios.ApplicationLayer.Scopes.MethodTraceScope
  • added Pantheios.Testing library that provides the Pantheios.Testing.Services.RecordingService service

Releases on Nuget:


Saturday, June 6, 2020

Expanding the Stock Severity Levels

Original Severity Levels

The original Pantheios C/C++ library defined eight severity levels (and their integral values and string forms), as follows:


Manifest constant Integral value String form (1)
PANTHEIOS_SEV_EMERGENCY 0 "Emergency"
PANTHEIOS_SEV_ALERT 1 "Alert"
PANTHEIOS_SEV_CRITICAL 2 "Critical"
PANTHEIOS_SEV_ERROR 3 "Error"
PANTHEIOS_SEV_WARNING 4 "Warning"
PANTHEIOS_SEV_NOTICE 5 "Notice"
PANTHEIOS_SEV_INFORMATIONAL 6 "Informational"
PANTHEIOS_SEV_DEBUG 7 "Debug"

1. as obtained by pantheios_getStockSeverityString()

For anyone familiar with Syslog, these level names and values will be very familiar, as they exactly match:


Manifest constantIntegral value
Emergency0
Alert1
Critical2
Error3
Warning4
Notice5
Informational   6
Debug7

Experiences and Evolution

However, over the decade and a half of use of Pantheios (and its other language bindings - .NET, Ruby, etc.) experience and further theoretical work has raised a number of issues, as discussed throughout the rest of this post.

"Error" -> "Failure"

As discussed in a 2013 instalment of Quality Matters in ACCU's Overload, the term error is fraught with ambiguity: almost the whole industry uses the term incorrectly to mean a failure - "throw an error" - whereas an error is something that a human does in designing or building a system.

So, we've started to use the term failure instead. Newer Pantheios projects, such as Pantheios.Ruby, already make use of this term, and the extant projects will evolve in that direction over the coming months.

"Emergency" -> "Violation"

Just as the term error is ambiguous so is the term bug. The issues around both terms are dealt with more formally in Contract Programming / Design by Contract, which is beyond the scope of this post, but you can read up on it in Meyer's Object-Oriented Software Construction or Wilson's Imperfect C++.

Contract Programming makes a formal distinction between a failure of a system and a violation of a software contract. When a contract violation occurs the program has behaved in contradiction to its design so by definition it cannot "correct". The only sensible course of action in this case is to (attempt to) terminate the process, yielding control to the operating environment in the hope that the process <-> operating system abstraction can handle the situation.

Over the lifetime of Pantheios we have taken to using the Emergency severity level exclusively for contract violations, and are starting to use the name Violation instead. Again, newer projects already have this, and we're evolving others.

Later in this post we will discuss the meanings we ascribe to the severity levels.

More debug levels

Extensive use in recent years of the Pantheios C/C++ library into sophisticated commercial C++ codebases with precise runtime controls over enabling/disabling severity levels has taught us that having a single debug level is insufficient. For example, when handling network traffic it is useful to be able to control the logging of the number of bytes received on a channel separate to the logging of all of those bytes.

As such, we've tended to have a number of debugging levels: Debug0Debug1, and so on, where each level involves deeper call levels and/or more extensive logged information. In the commercial projects we've provided custom severities with such additional levels, but plan to evolve the main Pantheios project along these lines soon.

"Trace"

There is one (new) particular debugging level that stands out from the others: Trace. This is useful for tracing function/method entry (including arguments received) and exit. Because tracing can lead to heavy amounts of logging, we have found it useful to give it a separate debugging level.

Again, Pantheios.Ruby has this level already, and other projects will be getting it soon. The current tracing facilities provided by the Pantheios C/C++ library defined in pantheios/trace.h  will be adjusted to use the new severity level.

"Benchmark"

Finally, there is one (new) level that has emerged, for the exclusive purpose of benchmarking.

Pantheios.X 202x levels

Hence, what we might call the 202x levels for all Pantheios projects has emerged, as follows:

Level name String form Former name Integral value
Violation "Violation" Emergency 1
Alert "Alert" Alert 2
Critical "Critical" Critical 3
Failure "Failure" Error 4
Warning "Warning" Warning 5
Notice "Notice" Notice 6
Informational "Informational" Informational 7
Debug0 "Debug-0" Debug 8
Debug1 "Debug-1" - 9
Debug2 "Debug-2" - 10
Debug3 "Debug-3" - 11
Debug4 "Debug-4" - 12
Debug5 "Debug-5" - 13
Trace "Trace" - 14
Benchmark "Benchmark" - 15


These levels already exist within Pantheios.Ruby and will be included in the next version of the Pantheios C/C++ library (albeit with a compile-time option to use the prior levels) and in the next release of Pantheios.NET. (Each of these is likely to be within a month or two of this post.)

Value changes

As well as the addition of new levels, and the renaming of two extant levels, you may also note that the values of the levels corresponding to the original eight have changed. Specifically, they are no longer 0-based, but are now 1-based. The reasons for this are:

  • to help catch programming errors in some languages where a default-initialised level variable/parameter would be interpreted as Violation(/Emergency); and
  • to allow sophisticated front-ends to have an "all levels" value to facilitate group control. (We'll discuss this more fully in a future post.)
Note that all levels still fall within 4-bits, so the extended severity level information mechanism is still supported by the the Pantheios C/C++ library. (See this post about extended severity information. Also, the next release will include C and C++ examples of how to use extended severity information.)

Ascribing meaning to levels

A previous post on this blog discusses the meanings we tend to ascribe to the 8 Syslog-derived levels. Naturally, if we're expanding the stock levels we need to take care to update this picture. In summary, we can say that we interpret them as follows, and offer it as a guide to you:


Level Meaning
Violation Used exclusively in the attempt to record a contract violation
Alert Used to report a practically-unrecoverable runtime condition, i.e. one that will require imminent process termination but is not a fault
Critical Serious failure to achieve normative behaviour
Error Failure to achieve normative behaviour
Warning Warning condition
Notice Information to be logged in normal operation, meaning when there has been no need to increase the logging information above a normal condition
Informational Logging information that is useful when actively monitoring the health of a system, but that is not necessarily displayed as part of "normal" operation 
Debug-0 Highest conceptual level and/or most terse form of debugging logging
Debug-1 Next lower conceptual level and more verbose than Debug-0
Debug-2 Next lower conceptual level and more verbose than Debug-1
Debug-3 Next lower conceptual level and more verbose than Debug-2
Debug-4 Next lower conceptual level and more verbose than Debug-3
Debug-5 Lowest conceptual level and most verbose form of debugging logging
Trace Used exclusively to record function/method entry/exit (along with, optionally, function arguments)
Benchmark Used exclusively to emit terse statements - usually only a literal string, to cause least impact to system performance - for the express purpose of tracking system performance

Friday, June 5, 2020

Custom Severity Levels with Pantheios C/C++ project

Been a bit of demand for this one, so here we go, in as few words as possible.

Stock Severity Levels

Out of the box, up to the present version (1.0.1 beta 218) the Pantheios C/C++ library comes with eight stock severity levels, as follows:


Manifest constantIntegral valueString form (1)
PANTHEIOS_SEV_EMERGENCY0"Emergency"
PANTHEIOS_SEV_ALERT1"Alert"
PANTHEIOS_SEV_CRITICAL2"Critical"
PANTHEIOS_SEV_ERROR3"Error"
PANTHEIOS_SEV_WARNING4"Warning"
PANTHEIOS_SEV_NOTICE5"Notice"
PANTHEIOS_SEV_INFORMATIONAL   6"Informational"   
PANTHEIOS_SEV_DEBUG7"Debug"

1. as obtained by pantheios_getStockSeverityString()

These are defined in the file pantheios/pantheios.h and are available for both C and C++ compilation units. Additionally, for C++ compilation units the following pseudo-constants are available:


C++ pseudo-constant (2)Integral ValueCorresponding constant
pantheios::emergency0PANTHEIOS_SEV_EMERGENCY
pantheios::alert1PANTHEIOS_SEV_ALERT
pantheios::critical2PANTHEIOS_SEV_CRITICAL
pantheios::error3PANTHEIOS_SEV_ERROR
pantheios::warning4PANTHEIOS_SEV_WARNING
pantheios::notice5PANTHEIOS_SEV_NOTICE
pantheios::informational   6PANTHEIOS_SEV_INFORMATIONAL
pantheios::debug7PANTHEIOS_SEV_DEBUG

2. these are defined 

These C++-only pseudo-constants also facilitate easily the use of extended severity information, but that's a topic for another time.

The string forms' C-style strings and string lengths are obtained, respectively, via the pantheios_getStockSeverityString() and pantheios_getStockSeverityStringLength() API functions.

Custom Severity Levels

If you want to use a different set of severities, then you must do the following:
  1. #define the preprocessor symbol PANTHEIOS_NO_STOCK_LEVELS;
  2. link to a back-end of your own that understands your custom severity levels.

Here's a C-program illustrating how to do this (with as much detail as possible elided for brevity):



#define PANTHEIOS_NO_STOCK_LEVELS /* suppress stock severity levels */

#include ⟨pantheios/pantheios.h⟩

/* some log4j-like levels */

enum CustomSeverityLevel_t
{
  CSL_Fatal = 0,
  CSL_Error = 3,
  CSL_Warn  = 4,
  CSL_Info  = 6,
  CSL_Debug = 7,
};

/* the application */

int main()
{
  int retCode = EXIT_SUCCESS;
  int res;

  res = pantheios_init();

  if(0 != res)
  {
    fprintf(stderr, "Failed to initialise the Pantheios libraries: %s\n", pantheios_getInitCodeString(res));

    retCode = EXIT_FAILURE;
  }
  else
  {
    pantheios_logputs(CSL_Info, PANTHEIOS_LITERAL_STRING("pantheios_logputs() can output a single C-style string"));


    . . . //rest of program logic, including more pantheios_logputs() / pantheios_logprintf() statements


    pantheios_uninit();
  }

  return retCode;
}

/* the custom back-end */

#include ⟨pantheios/backend.h⟩
#include ⟨pantheios/init_codes.h⟩
#include ⟨pantheios/strdup.h⟩

PANTHEIOS_CALL(int) pantheios_be_init(
  PAN_CHAR_T const* processIdentity
, void*             reserved
, void**            ptoken
)
{
  ((void)reserved);

  *ptoken = pantheios_util_strdup_nothrow(processIdentity);

  return (NULL == *ptoken) ? PANTHEIOS_INIT_RC_OUT_OF_MEMORY : PANTHEIOS_INIT_RC_SUCCESS;
}

PANTHEIOS_CALL(void) pantheios_be_uninit(void *token)
{
  pantheios_util_strfree((PAN_CHAR_T*)(token));
}

PANTHEIOS_CALL(int) pantheios_be_logEntry(  
  void*             feToken
, void*             beToken
, int               severity
, PAN_CHAR_T const* entry
, size_t            cchEntry
)
{
  PAN_CHAR_T const* sevString;
  PAN_CHAR_T const* processId = (PAN_CHAR_T const*)(beToken);

  ((void)feToken);

  switch (severity)
  {
  case CSL_Fatal:
    sevString = PANTHEIOS_LITERAL_STRING("Fatal");
    break;
  case CSL_Error:
    sevString = PANTHEIOS_LITERAL_STRING("Error");
    break;
  case CSL_Warn:
    sevString = PANTHEIOS_LITERAL_STRING("Warn");
    break;
  case CSL_Info:
    sevString = PANTHEIOS_LITERAL_STRING("Info");
    break;
  case CSL_Debug:
    sevString = PANTHEIOS_LITERAL_STRING("Debug");
    break;
  default:
    sevString = PANTHEIOS_LITERAL_STRING("");
    break;
  }

#ifdef PANTHEIOS_USE_WIDE_STRINGS
  return fwprintf(
        stderr
      , L"[%s; %s]: %.*s\n"
      , processId
      , sevString
      , (int)cchEntry, entry
      );
#else /* ? PANTHEIOS_USE_WIDE_STRINGS */
  return fprintf(
        stderr
      , "[%s; %s]: %.*s\n"
      , processId
      , sevString
      , (int)cchEntry, entry
      );
#endif /* PANTHEIOS_USE_WIDE_STRINGS */
}