Service

Service

A service is a bundled software package with associated management scripts that handle the configuration, starting, stopping, and ongoing management of the application. Services are where the behavior of enStratus can be extended and customized to perform as desired.

Understanding services is critical to successful automation.

NOTE: Services are also referred to as service images because they are in some ways analogous to the way machine images are handled.

Service Scripts and Agent Scripts

Agent Script	Purpose	Override	Service Script
backupDataSource	Initiates service backup	N	enstratus-backupDataSource
backupService	Initiates service backup	Y	enstratus-backupService
checkService		N	N/A
configureService	Initiates service specific configuration	N	enstratus-configure
grantDatabaseAccess	Initiates a service specific script for granting database access	N	enstratus-dbgrant
installDataSource	Initiates database specific scripts for adding the data source	N	enstratus-installds
installService	Installs service	N	N
scaleCheck	Returns status code that is used in voting for scaling	Y	N
startProxy	Determines proxy method and calls proxy scripts	N	$proxyType-startProxy
startService	Initiates start service script	N	enstratus-start
stopProxy	Determines proxy method and calls proxy script	N	$proxyType-stopProxy
stopService	Initiates stop service script	N	enStratus-stop

Service Structure

The structure of an enStratus is mostly free form. enStratus automation was designed to allow for as much freedom as possible when designing and implementing deployments. Some specific requirements will be described here.

Typically, a service image is a zip file that contains a directory hierarchy. The directory hierarchy can vary depending on the service image. We’ll start by examining a generic service image, which will be useful in pulling out some specific functions. Following this discussion, we’ll move to a specific and more complex example for MySQL.

Our generic service image will have three directories, bin/, cfg/, and www/. These directories will be arranged as shown in the following table:

Directory	Contents
bin/	enstratus-configure enstratus-start enstratus-stop
cfg]	<empty>
www	index.html

Generally, service images operate with three basic scripts, enstratus-configure, enstratus-start, and enstratus-stop. The scripts written in any language (so long as the language is installed upon the server itself) and can be written to accomplish just about anything you desire.

The three basic scripts are special because they are ”hooks” that enStratus uses to configure, start, and stop services. During the start of an enStratus deployment, the enStratus agent will be used to kick off service related events. The following table and diagram are provided to help clarify.

Agent Script	Service Script
/enstratus/bin/configureService	enstratus-configure
/enstratus/bin/startService	enstratus-start
/enstratus/bin/stopService	enstratus-stop

Of these three scripts, enstratus-configure is the most important. Note that in /enstratus/bin/- configureService agent script calls the service script with two arguments. The first argument is the serviceId, which is just a string of the form a1234 which enStratus assigns to all services. The second argument is a properties file, an example of which will be shown in this section.

It will be useful now to examine the process by which enStratus starts and configures a server during deployment start.

Clustered Server Launch Process

Click to Enlarge

There’s a lot of information going on in that previous diagram. For now, we’ll pick up the process in step 4. At this step, we assume that a server and the enStratus agent are operational. Also, a storage volume is sometimes attached for housing the application. The next thing enStratus does is call the installService script located in /enstratus/bin on the server itself. This script unzips the service image, and installs it to the appropriate service directory, in this case it is /mnt/services/a1234.

Next comes the critical step. In step 5, the enStratus provisioning server calls the agent script called /enstratus/bin/configureService. This script does a few things, but most importantly it calls a service script /mnt/services/a1234/bin/enstratus-configure with two arguments. The first argument is the serviceId, in this case a1234. The second argument is more interesting, it is a configuration file called enstratus.cfg.

enstratus.cfg

An example of an enstratus.cfg file is provided here:

1 # enStratus service configuration file
2 # Generated Fri Feb 05 01:29:06 UTC 2010
3 [enstratus]
4 serverId=1234
5 serviceId=a1234
6 name=MySQL
7 protocol=TCP
8 customerId=c001
9 publicPort=3306
10 privatePort=3306
11 # Proxy port is the same thing as private port;
12 # it exists for backwards compatibility
13 proxyPort=3306
14 primaryAddress=localhost
15 addresses=localhost
16 userId=mysql
17
18 runAsUser=mysql
19 rootUser=custAdmin
20 rootPassword=s33kr3t
21
22 # Custom configuration data
23 [custom]
24 user1=pa55w0rd
25 user2=winn3r
26
27 access_key=ASDFASDFASDFASDFASDF
28 secret_key=ADSFADSFADSFADSFADSFADSFADSFADSFADSFADSF

Each enstratus.cfg file is broken into two sections

Section	Origin
[enstratus]	Deployment configuration options
[custom]	User input from custom field

The purpose of this file is to pass information to a service both at start time and whenever a service upon which another service depends changes. For example, if a web service depends on a database service and there is a scaling event to include another web server, there will be two enstratus.cfg files generated. One will be passed to the web service with information about where it can find the database service. One will be passed to the database service with information about the web service, such as the private IP address of the web server which is ultimately processed by the mysqlGrant script as part of the MySQL service image.

In this way, enStratus continually monitors and passes information to dependent tiers of servers when critical information changes.

This enstratus.cfg file is an example of content that would be passed to a MySQL Database server during start. An example of an enstratus.cfg file that would be passed to an application service that depends on a database service is shown here:

1 # enStratus service configuration file
2 # Generated Wed Sep 08 19:02:58 UTC 2010
3 [enstratus]
4 serverId=20789
5 serviceId=a3723
6 name=Drupal
7 customerId=c100
8 #BEGIN DEPRECATED SECTION
9 protocol=TCP
10 publicPort=80
11 privatePort=80
12 #Proxy port is the same thing as private port; it exists for backwards compatibility
13 proxyPort=80
14 #END DEPRECATED SECTION
15 port.80.protocol=TCP
16 port.80.publicPort=80
17 port.80.privatePort=80
18 port.80.dnsNames=examplesite.com
19 port.443.protocol=TCP
20 port.443.publicPort=443
21 port.443.privatePort=80
22 port.443.dnsNames=examplesite.com
23 #BEGIN DEPRECATED SECTION
24 primaryAddress=examplesite.com
25 addresses=
26 #END DEPRECATED SECTION
27 userId=c100
28 dataSources=drupal
29 dataSource.drupal.read=10.212.91.212
30 dataSource.drupal.write=10.212.91.212
31 dataSource.drupal.serviceUser=drupal
32 dataSource.drupal.servicePassword=trustno1
33
34 # Custom configuration data
35 [custom]


enstratus.cfg security

NOTE: The enstratus.cfg file is quite temporal. It is only present on the system long enough to be passed as an argument to enstratus-configure. Once that action is completed, enStratus securely erases enstratus.cfg.

This process should give you confidence that it is okay to place sensitive information such as user names and passwords into the enstratus.cfg file via the custom field.

enstratus-configure

One of the primary purposes of the enstratus-configure script is to process the parameters passed in via the enstratus.cfg file. As previously mentioned, the enstratus-configure script can be written in the preferred language of the author, provided the language is supported by the machine upon which the script is executed.

This script is an example of an enstratus-configure written in the shell programming language. It is not very complex and again its primary function is to parse the enstratus.cfg file and use those variables to configure a service, in this case a web service for the Drupal application.

1 #!/bin/bash
2
3 LOGGER=/enstratus/bin/log
4 logTag="enstratus-configure"
5
6 $LOGGER -t "$logTag Entering enstratus-configure."
7
8 serviceId=$1
9 configFile=$2
10
11 serverId=$( grep serverId $configFile | cut -f 2 -d= )
12 serviceId=$( grep serviceId $configFile | cut -f 2 -d= )
13 name=$( grep name $configFile | cut -f 2 -d= )
14 dataSources=$( grep dataSources $configFile | cut -f
15
16 readIP=$( grep "read" $configFile | cut -f 2 -d= )
17 writeIP=$( grep "write" $configFile | cut -f 2 -d= )
18 serviceUser=$( grep serviceUser $configFile | cut -f
19 servicePass=$( grep servicePassword $configFile | cut -f 2 -d= )
20
21 publicPort=$( grep publicPort $configFile | cut -f 2 -d= )
22 privatePort=$( grep privatePort $configFile | cut -f 2 -d= )
23
24 domain=$( grep primaryAddress $configFile | cut -f 2 -d= )
25
26 #Testing only, do not use in production.
27 #This copies over the enstratus.cfg file to /mnt/tmp
28 sudo cp cfg/enstratus.cfg /mnt/tmp
29
30 #Clear out any sites-enabled
31 sudo rm -f /etc/init.d/apache2/sites-enabled/* > /dev/null 2>&1
32 sudo /etc/init.d/apache2 stop > /dev/null 2>&1
33 sleep 3
34
35 #This is a bit of a hack to make drupal happy at first start.
36 sudo mkdir /mnt/services/$serviceId/www/sites/default/files > /dev/null 2>&1
37 sudo chmod 777 /mnt/services/$serviceId/www/sites/default/files > /dev/null 2>&1
38
39 sudo sed -i "s/DocumentRoot \/var\/www/DocumentRoot
40 \/mnt\/services\/$serviceId\/www/" /etc/apache2/sites-available/default
41 sudo sed -i "s/Directory \/var\/www/Directory \/mnt\/services\/$serviceId\/www/"
42 /etc/apache2/sites-available/default
43
44 #Logic to handle change in IP address of MySQL server in the event it fails.
45 count=$( grep -c $servicePass /mnt/services/$serviceId/www/sites/default/settings.php
46 )
47
48 if [ "$count" = 0 ]; then
49         $LOGGER -t "$logTag This is a clean start, creating MySQL connection string
50 from scratch."
51         sudo sed -i
52 "s/MYDB/mysql:\/\/$serviceUser:$servicePass@${readIP}\/$dataSources/"
53 /mnt/services/$serviceId/www/sites/default/settings.php
54 else
55         $LOGGER -t "$logTag Fixed settings.php for new DB server."
56         $LOGGER -t "$logTag IP of recoverd DB: $readIP"
57 #Make a backup of the settings.php file before doing this
58         sudo cp /mnt/services/$serviceId/www/sites/default/settings.php{,.bak} >
59 /dev/null 2>&1
60 #Now, replace the old IP with the new IP of the recovered DB server
61         sed -i "s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/$readIP/"
62 /mnt/services/$serviceId/www/sites/default/settings.php
63 fi
64 #Finally, restart apache
65 sudo ln -s /etc/apache2/sites-available/default /etc/apache2/sites-enabled/default >
66 /dev/null 2>&1
67 sudo a2ensite default > /dev/null 2>&1
68
69 sudo /etc/init.d/apache2 restart > /dev/null 2>&1


This script is an example of an enstratus-configure written in the shell programming language. It is not very complex and again its primary function is to parse the enstratus.cfg file and use those variables to configure a service, in this case a web service for the Drupal application.

Here are the key points to remember about the enstratus-configure script:

1. It may be written in any language, provided that the language is installed upon the server on which the script is executed.
   
   
2. It is passed two arguments:
   
   (a) serviceId (ex: a1234)
   (b) enstratus.cfg
   
   
3. It’s primary purpose should be to process the information contained in enstratus.cfg.
   
   
4. It is called:
   
   (a) On every server in each server group on deployment start.
   (b) On every dependent server during a scaling or auto-recovery event.
   
   

enstratus-start

The second critical script that should be present in every service image is the enstratus-start script. The purpose of this script is to call logic to start the service. The service can be an application, a database, or something else. In the figure above, step 6 is the step where this script is called. Once the enstratus-configure script has completed successfully, the next thing enStratus does is to call the agent script /enstratus/bin/startService. The startService script does little more than call /mnt/services/$serviceId/bin/enstratus-start with one argument: the serviceId.

An example enstratus-start script is shown here:

1 #!/bin/bash
2
3 LOGGER=/enstratus/bin/log
4 logTag="enstratus-start"
5
6 $LOGGER -t "$logTag Calling enstratus start. Starting Apache"
7
8 sleep 3
9 sudo /etc/init.d/apache2 restart
10
11 if[ $? !=0 ];then
12     $LOGGER -t "$logTag Apache start failed."
13 else
14     $LOGGER -t "$logTag Apache start successful."
15 fi


In this case, the script is written in Bash, and does nothing more than start the Apache web server and report on the outcome.

Here are some key points to remember about the enstratus-start script:

1. It may be written in any language, provided that the language is installed upon.
   
   
2. It is passed one argument:
   (a) serviceId (ex:a1234)
   
   
3. It should contain logic required to start the service.
   
   
4. It is called:
   (a) On every server in each server group on deployment start.
   (b) On every dependent server during a scaling or auto-recovery event.

enstratus-stop

The last of the three key scripts that should be present in every service image is the enstratus- stop script. The purpose of this script is to call logic to stop the service. The service can be an application, a database, or something else. The enstratus-stop script is called whenever a scaling event occurs in a server group where servers are terminated.

When a server is targeted for termination by enStratus, enStratus will first call the agent script /enstratus/bin/stopService, which does little more than call /mnt/services/$serviceId/bin/enstratus- stop with one argument: the serviceId.

An example enstratus-stop script is shown here:

1 #!/bin/bash
2
3 LOGGER=/enstratus/bin/log
4 logTag="enstratus-stop"
5
6 $LOGGER -t "$logTag Calling enstratus stop. Starting Apache"
7
8 sudo /etc/init.d/apache2 stop
9
10 if[ $? !=0 ] ; then
11         $LOGGER -t "$logTag Apache stop failed."
12 else
13     $LOGGER -t "$logTag Apache stop successful."
14 fi


In this case, the script is written in Bash, and does nothing more than stop the Apache web server and report on the outcome.

Here are some key points to remember about the enstratus-stop script:

1. It may be written in any language, provided that the language is installed upon.
   
   
2. It is passed one argument:
   (a) serviceId (ex:a1234)
   
   
3. It should contain logic required to stop the service.
   
   
4. It is called:
   (a) When a server is being terminated as part of an auto-scaling event.
   
   

cfg directory

The other required directory in a service image is the cfg/ directory. The purpose of this directory is to house configuration files such as the enstratus.cfg file. We’ll see in a moment that it is also used as a storage area for configuration files that may be critical to the operation of a specific application such as a MySQL database.

Summary

These are the basic points of a service image in enStratus. Each service image should contain a bin/ and cfg/ directory. Inside the bin/ directory should be three scripts. enstratus-start, enstratus-configure, and enstratus-stop.

Next, let’s look at a more complex service image, the MySQL service.

back to top