The EFT.log file; HTTP Request Logging

THE INFORMATION IN THIS ARTICLE APPLIES TO:

• EFT 
• EFT Arcus

EFT v8.0 and later store Advanced Properties in a JSON file. When you upgrade from EFT v7.4.x to EFT v8, the non-default settings that you have defined in the registry will be added to the Advanced Properties file during upgrade. (Default settings become part of the EFT configuration files.) For a more on how to use advanced properties, and a spreadsheet of the advanced properties, please refer to the "Advanced Properties" topic in the help for your version of EFT.

DISCUSSION

EFT Server can be instructed to log specific or all HTTP request headers. When this following advanced property value is set to 1, all HTTP request headers will be logged:

"log_request"=dword:1

1 = enable "all request header" logging

0 = disable "all request header" logging [default]

In version 8 and later:

Add the name:value pair to the AdvancedProperties.JSON file in EFT's \ProgramData\ directory as described in the "Advanced Properties" topic in the online help for your version of EFT.

{
"log_request": 1
}

In versions prior to version 8, add the property to the registry.

32-bit OS:

[HKEY_LOCAL_MACHINE\SOFTWARE\GlobalSCAPE Inc.\EFT Server 4.0\EFTClient]

64-bit OS:

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\GlobalSCAPE Inc.\EFT Server 4.0\EFTClient]

For this registry key to work, a file called logging.cfg in the EFT Server installation folder (e.g., (C:\Documents and Settings\All Users\Application Data\GlobalSCAPE\EFT Server Enterprise\logging.cfg), must be configured to log at the DEBUG level. With this logging file, EFT Server logs requests that require authentication and those that cause session-checking failures. The information is logged to a file named EFT.log in the EFT installation folder (e.g., C:\Documents and Settings\All Users\Application Data\GlobalSCAPE\EFT\). If you want to save EFT.log to a different location, change the reference at the bottom of the logging.cfg file from the following location to the location you prefer:

log4cplus.appender.R.File=${AppDataPath}\EFT.log

(Note: On Windows 2008 and later, Application Data files for all users are in a hidden folder named %systemroot%\ProgramData instead of under Documents and Settings\All Users\Application Data.)

• For details of logging HA activities, jump to High Availability Logging, below.

• For information regarding error codes in the EFT.log, please refer to the MSDN article at http://msdn.microsoft.com/en-us/library/windows/desktop/ms681381%28v=vs.85%29.aspx. For example, error code "5 (0x5)" indicates that access is denied.

It is not necessary to restart the EFT server service after editing the logging.cfg file.

Logger Hierarchy

All loggers inherit from the root logger; i.e., this is the default level that applies to all loggers:

log4cplus.rootLogger=INFO, RootFileAppender

Each logger's level can be set independently. Children inherit their parents setting unless set explicitly. For example, you can set Events and its children to a different level than the others:

log4cplus.logger.Events=TRACE

To enable trace level folder monitor logging for the site "My Site" the logger entry would be:

log4cplus.logger.Events.FolderMonitor.My_Site=TRACE

To turn on logging for a specific event rule, you can append the event rule name (spaces replaced with underscores _) after the site name. The logger name will have this format:

log4cplus.logger.Events.[optional event sub logger].[site 
 name].[event rule name]

For example, to enable trace level logging for the folder monitor rule "My Folder Monitor" for the site "My Site" the logger entry would be:

log4cplus.logger.Events.FolderMonitor.My_Site.My_Folder_Monitor=TRACE

• Since logger names are case sensitive, the case of Site and Event Rule names used as loggers must match their use in EFT.

• If Site or Event Rule names contain spaces they must be replaced with underscores when used in the logger entry.

• Not all event logging entries support Site or Event Rule level logging.

The logger hierarchy in EFT includes the following events:

#log4cplus.logger.Administrator=TRACE
#log4cplus.logger.Administrator.Permissions=TRACE
#log4cplus.logger.AdministratorSupport=TRACE
#log4cplus.logger.AdvancedProperties=INFO
#log4cplus.logger.ARM=TRACE
#log4cplus.logger.ARM.Queue=TRACE
#log4cplus.logger.AS2=TRACE
#log4cplus.logger.AUD.Read=TRACE
#log4cplus.logger.AUD.Write=TRACE
#log4cplus.logger.AuthManager=TRACE
#log4cplus.logger.AuthManager.RADIUS=TRACE
#log4cplus.logger.AuthManager.RADIUS.Packet=TRACE
#log4cplus.logger.AuthManager.RSA=TRACE
#log4cplus.logger.AWE=TRACE
#log4cplus.logger.Backup=TRACE
#log4cplus.logger.CFG.Read=TRACE
#log4cplus.logger.CFG.Write=TRACE
#log4cplus.logger.ClientManager=TRACE
#log4cplus.logger.ClientTransfers=TRACE
#log4cplus.logger.CmdAccess=TRACE
#log4cplus.logger.Common=TRACE
#log4cplus.logger.DMZSupport=TRACE
#log4cplus.logger.Events=TRACE
#log4cplus.logger.Events.AS2=TRACE
#log4cplus.logger.Events.Client=TRACE
#log4cplus.logger.Events.Conn=TRACE
#log4cplus.logger.Events.FolderMonitor=TRACE
#log4cplus.logger.Events.FS=TRACE
#log4cplus.logger.Events.Server=TRACE
#log4cplus.logger.Events.Site=TRACE
#log4cplus.logger.Events.Clustered=TRACE
#log4cplus.logger.Events.ContentIntegrityControl=TRACE
#log4cplus.logger.Events.Workspaces=TRACE
#log4cplus.logger.Events.FolderActions=TRACE
#log4cplus.logger.Events.FileActions=TRACE
#log4cplus.logger.Events.CompressDecompressActions=TRACE
#log4cplus.logger.Events.CompressDecompressServer=TRACE
#log4cplus.logger.Events.WebServices=TRACE
#log4cplus.logger.Events.S3=TRACE
#log4cplus.logger.Events.AzureStorage=TRACE
#log4cplus.logger.Events.AzureDataLakeStorageGen2=TRACE
#log4cplus.logger.Events.GoogleDrive=TRACE
#log4cplus.logger.Events.Cloud=TRACE
#log4cplus.logger.Events.PowerShell=TRACE
#log4cplus.logger.Events.Other=TRACE
#log4cplus.logger.FileSystem=TRACE
#log4cplus.logger.FTP=TRACE
#log4cplus.logger.HTTP=TRACE
#log4cplus.logger.HTTP.Handler=TRACE
#log4cplus.logger.HTTP.SessionManager=TRACE
#log4cplus.logger.IPAccess=TRACE
#log4cplus.logger.PathManager=TRACE
#log4cplus.logger.PGP.Adapter=TRACE
#log4cplus.logger.Registration=TRACE
#log4cplus.logger.Reporting=TRACE
#log4cplus.logger.Reports=TRACE
#log4cplus.logger.Server.Startup=TRACE
#log4cplus.logger.Server.Stop=TRACE
#log4cplus.logger.Service=TRACE
#log4cplus.logger.SFTP=TRACE
#log4cplus.logger.SMTP=TRACE
#log4cplus.logger.SSL=TRACE
#log4cplus.logger.Status Viewer=TRACE
#log4cplus.logger.Timer=TRACE
#log4cplus.logger.Cluster=TRACE
#log4cplus.logger.Cluster.SharedFiles=TRACE
#log4cplus.logger.Cluster.ChangeQueue=TRACE
#log4cplus.logger.Cluster.SyncStatus=TRACE
#log4cplus.logger.Workspaces=TRACE
#log4cplus.logger.Workspaces.Invite=TRACE
#log4cplus.logger.SAMLSSO=TRACE
#log4cplus.logger.Cloud=TRACE
#log4cplus.logger.Admin.API=TRACE
#log4cplus.logger.Remote.Agent=TRACE
#log4cplus.logger.CAPTCHA=TRACE
#log4cplus.logger.EncryptedFolders=TRACE
#log4cplus.logger.User.API=TRACE
#log4cplus.logger.SecretsModule=TRACE
#log4cplus.logger.SMS=TRACE (This logger only displays information when the SMS provider is correctly configured.)
#log4cplus.logger.UploadQuota=TRACE	

log4cplus.logger.<name_of_event_to_log>=<log_level>

For example:

log4cplus.logger.Server.Startup=TRACE

Event Rule Loggers for Specific Sites and Rules

To turn on event rule logging for a specific site you can append the site name (spaces replaced with '_') to the name of a standard event logger. The logger name will have this format:

log4cplus.logger.Events.[optional event sub logger].[site name]

For example, to enable trace level folder monitor logging for the site "My Site" the logger entry would be:

log4cplus.logger.Events.FolderMonitor.My_Site=TRACE

To turn on logging for a specific event rule, you can append the event rule name (spaces replaced with '_') after the site name. The logger name will have this format:

log4cplus.logger.Events.[optional event sub logger].[site name].[event rule name]

For example, to enable trace level logging for the folder monitor rule "My Folder Monitor" for the site "My Site" the logger entry would be:

log4cplus.logger.Events.FolderMonitor.My_Site.My_Folder_Monitor=TRACE

SQL Lite Statement Auditing

log4cplus.logger.SQLite=WARN
#log4cplus.logger.SQLite=TRACE, SQLiteFileAppender
#log4cplus.logger.SQLite.Prepare=TRACE, SQLiteFileAppender
#log4cplus.logger.SQLite.Execute=TRACE, SQLiteFileAppender
#log4cplus.logger.SQLite.Deallocate=OFF, SQLiteFileAppender
#log4cplus.additivity.SQLite=false
#Log to a file called "SQLite.log" in the ProgramData folder.
#log4cplus.appender.SQLiteFileAppender=log4cplus::RollingFileAppender
#log4cplus.appender.SQLiteFileAppender.File=${AppDataPath}\SQLite-${COMPUTERNAME}.log
#log4cplus.appender.SQLiteFileAppender.MaxFileSize=20MB
#log4cplus.appender.SQLiteFileAppender.MaxBackupIndex=5
#log4cplus.appender.SQLiteFileAppender.layout=log4cplus::PatternLayout
#log4cplus.appender.SQLiteFileAppender.layout.ConversionPattern=[%d{%y-%m-%d %H:%M:%S}][%-5p][%c{2}] - %m%n

Cloud Logging

To enable AWS-related logging, in logging.cfg, look for the following section, and change OFF to ON:

#Turn off AWS logging for traditional deployments
#log4cplus.logger.Cloud.AWS=OFF

File System Logging

A File System Logging appender can be used to log file system activities to a separate file for debug diagnostics, as shown below.

#log4cplus.appender.FileSys=log4cplus::RollingFileAppender
#log4cplus.appender.FileSys.File=${AppDataPath}\EFT-FileSystem.log
#log4cplus.appender.FileSys.MaxFileSize=20MB
#log4cplus.appender.FileSys.MaxBackupIndex=5
#log4cplus.appender.FileSys.layout=log4cplus::TTCCLayout
#log4cplus.logger.FileSystem=DEBUG, FileSys
 #log4cplus.additivity.FileSystem=false

DEBUG will capture all activity and time taken.

It is not recommended to use TRACE, as this will create quite a bit of unusable chatter in the log.

Log Levels

EFT.log organizes logging levels as a hierarchy: TRACE, DEBUG, INFO, WARN, ERROR, FATAL, OFF. When you specify a logging level, each of the lower levels is also included. That is, if you specify DEBUG level, you also get INFO, WARN, ERROR, and FATAL logs. However, if you specify INFO, you will not log DEBUG or TRACE level activities.

The log reference to "Timeout" is not actually a problem, it is by design in our software. Our auditing mechanism periodically disconnects and reconnects to the database server in order to avoid complications of long-running open connections; in particular, drivers in Oracle 11g and earlier are notorious for having slow memory leaks that caused problems over time. Thus, our system has a timeout value (default to 180 seconds) where we disconnect and reconnect to ensure clean processing. The 180 seconds is currently hard-coded into EFT.

For example, these log entries are fully expected and do NOT indicate any error:

06-10-15 11:36:29,997 [1848] INFO ARM <SQL Queue Reader> 
 - Timeout: closing the database connection [timeout = L180 seconds]
  06-10-15 11:36:32,774 [1848] INFO ARM <SQL Queue Reader> 
 - Database connection closed. Reconnecting...
  06-10-15 11:36:32,852 [1848] INFO ARM <SQL Queue Reader> 
 - Reconnection successful

Appenders

Appenders determine where the output of the logging goes. Each logger can have more than one appender and inherits appenders from parents by default. Appenders have an associated layout that determines the content of the log lines. EFT uses a RollingFileAppender with the TTCCLayout. With this layout, the log contains the name of the logger, date/time, thread id, the log line itself, and other things.

The file logging.cfg in the EFT installation folder provides details of how EFT uses Log4Cplus. For more information about Log4Cplus, refer to http://log4cplus.sourceforge.net/docs/html/classlog4cplus_1_1PropertyConfigurator.html

The following appenders are enabled by default, under Main EFT Log File Appender:

log4cplus.appender.RootFileAppender=log4cplus::RollingFileAppender
log4cplus.appender.RootFileAppender.File=${AppDataPath}\EFT.log
# Each log file will grow up to 20MB in size
log4cplus.appender.RootFileAppender.MaxFileSize=20MB
# Once a log file reaches the maximum file size it will be renamed to a backup
# file. Up to 5 backup files will be kept.
log4cplus.appender.RootFileAppender.MaxBackupIndex=5
# The TTCCLayout outputs time, thread, logger, nested diagnostic context, and log line
log4cplus.appender.RootFileAppender.layout=log4cplus::TTCCLayout
log4cplus.appender.RootFileAppender.layout.DateFormat=%m-%d-%y %H:%M:%S,%q

High Availability Logging

This section of the logging.cfg is disabled (commented out) by default. Log HA activities to a separate file for debug diagnostics by enabling the following section of the file (at the bottom of the file):

#log4cplus.appender.HAAppender=log4cplus::RollingFileAppender
#log4cplus.appender.HAAppender.File=${AppDataPath}\EFT-HA.log
#log4cplus.appender.HAAppender.MaxFileSize=20MB
#log4cplus.appender.HAAppender.MaxBackupIndex=5       
#log4cplus.appender.HAAppender.layout=log4cplus::TTCCLayout
#log4cplus.appender.HAAppender.layout.DateFormat=%m-%d-%y %H:%M:%S,%q
#log4cplus.logger.Cluster=TRACE, HAAppender
#log4cplus.logger.Events.Clustered=TRACE, HAAppender

Review the logging.cfg file for examples.