How to start?

Some people ask me - how to start building a production system, and make sure that it will be reliable, scaleable and manageable when it will outgrow two cabinets in one location (a kind of my definition of a small system)?

Start with breaking down the large task to small pieces. The following are some examples of "small pieces":

• Select the OS platform and distribution to go with (if your R&D leaves you a choice)
• Select the hardware platform

• Select the load balancing solution
• Select the database platform (once again - if the R&D leaves you a choice)
• Select the central configuration management system
• Select the colocation facility(s) where you will host your equipment
• Select the Internet provider(s)
• Select the Out-Of-Band (OOB) channel to access your site
• Select the power management solution for your equipment
• Select the console or KVM solution to manage your devices
• Select the monitoring solution
• Select the remote VPN access solution
• Define the equipment naming convention
• Define the traffic routing schema
• Define the traffic load balancing schema
• Define your IP address space requirements
• Describe the main data flows in the system

All together the listed items will form a high level design of your future production environment.

Now go down to more detailed level:

• Define the location of the equipment in the cabinets
• Design the network subneting plan
• Define the configuration standards for all types of used equipment - servers, routers, switches, PDUs, console servers, etc
• Define the configuration standards for all used applications - in-house products, third-party products, OS built-in applications
• Define the low level design of the network: document all used power, network and console ports, all IP addresses
• Document all data flows in the network (required to configure firewall rules)

Once you will get all required equipment deployed at the data center you may start building the system.

Build the following services:

• Wiki documentation management service
• Local software repository
• Automated servers installation service (Kickstart/Jumpstart)
• Monitoring service
• Central configuration management service
• Local DNS service
• Remote VPN access service

You will also need to set expectations with your R&D - please see the post regarding this topic.

Document the following information:

• Vendors contact list
• Operations contact list + responsibilities
• R&D contact list + responsibilities
• Equipment inventory list

Formulate the following procedures:

• Change management procedure
• Operations on-call procedure
• Operations escalation procedure
• Password management policy
• Information security policy

Obviously, this is not a complete list, and I will revise it from time to time, and update with links to my new posts.

Cheers.