Operations requirements for in-house R&D products

This post will help you to define specific requirements from Operations to R&D for all in-house software provided for production deployment if you faced with in-house R&D products.

Once the requirements are modified to suit the specific environment, they should be discussed with your company's R&D manager, and approved by both parties. From the moment all new software products should compliant with the standard, and a strict timetable should be defined to fix all existing products (and expect that this will take some time).

Packaging and package names
All in-house software should be packaged using the standard packaging format supported by used operating system: RPM for RHEL/CentOS, DEB for Debian-like systems, etc.

While a proper software packaging adds some extra work for R&D, it provides a lot of benefits:

• Easy deployment in production and QA environments
• A possibility to manage local software repositories accessible by OS default package manipulations tools (like "yum" in RHEL)
• Simple audit of installed products and their versions
• Easy rollbacks for failed deployments
• Simple automation of all pre- and post-installation manipulations using the package pre- and post-install scripts

There should be a naming standard for all products delivered by R&D, and software packages should be  named with product names. For example, if the company name is FLR, it might be wise to add prefix "flr-" to all in-house products. Some examples of product (and package) names:

• flr-api
• flr-log-processor
• flr-data-export
• flr-proxy

Package file structure
I recommend to install all software which is not a part of the OS in directory "/opt" in individual sub-directories named like the package name. For example, if the package name is "ytp-crawler" the software files should be placed in directory "/opt/ytp-crawler" - so called application home directory.

I also recommend to place different application files to different sub-directories inside the application home directory depending on the files function:

• Configuration files should go to "etc/" directory; for example, "/opt/ytp-crawler/etc"
• Binary files and scripts should go to "bin/" directory; for example, "/opt/ytp-crawler/bin"
• Individual log files (if for a reason the application cannot use the OS standard "syslog" logging facility)  should go to "log/" directory; for example, "/opt/ytp-crawler/log"
• Application data files should be stored in "data/" directory; for example, "/opt/ytp-crawler/data"

Configuration files
While in the most of OS packages the configuration files are placed inside the packages, for in-house software package I would recommend to not pack configuration files inside the packages. The application still should be configured to look for configuration files inside the application-specific "etc" directory (like "/opt/ytp-crawler/etc"), but the configuration files should be managed using a centralized configuration management system like Puppet or Chef (I will discuss Puppet in a separate post).

It is still pretty convenient to place the examples of configuration files in "etc/" directory, and use suffix ".def" (from "default") to distinguish the examples from real configuration files. An example of sample configuration file name is "/opt/ytp-crawler/etc/crawler.conf.def" (and the real configuration file will be named "/opt/ytp-crawler/etc/crawler.conf").

Temporary files
I recommend to put application's temporary files in system-wide "/tmp" folder. On RHEL/CentOS systems the directory is automatically cleaned by "tmpwatch" tool (see "/etc/cron.daily/tmpwatch" file how the process is configured).

It is also possible that an application will use an application-specific temporary directory (for example, "/opt/flr-fetcher/tmp/"), but it will be required to manually take care about regular cleaning of the directory.

Run control scripts
In case if an application is designed to run in unattended mode for long period of time, than it should be accomplished with a proper run control script placed in the application's "bin" directory, and with a symbolic link placed in the system wide directory "/etc/init.d" for run control scripts.

Logging
As a rule all in-house applications should be designed to send application log messages via OS standard "syslog" facility - in this way it will be possible to easily route all application logs to a central syslog server, and monitor them using a log monitoring tool (I will talk about "rsyslog" and "logcheck" in a separate post).

If for a reason an application cannot use the "syslog" facility, than it should write its logs in directory "log/" inside the application home directory; for example, "/opt/flr-fetcher/log/fetcher.log".

Application logging is a quite important topic, and I will discuss it more in a separate blog post.

Have I missed something? Your comments are welcome!