Salesforce Continuous Integration

Continuous Integration of Salesforce metadata can be tricky but when working on any codebase with multiple developers it becomes hard to ensure that changes to the codebase don't break the build. For traditionally compiled languages this is less of a problem as the code is typically fully compiled during the development process. Salesforce is a bit different however as the code must be compiled on the server and a full compile of the code can be arduous. Often the most common issue is forgetting to update a @test class along with changes to the class that its unit tests are targeted for however other classes may be dependent on the old version of the code as well. Continuous integration servers like Jenkins provide a solution for this by automatically building the code as it is changed in the source code repository. For Salesforce, we need to be able to compile this code in an SFDC instance. Although it would be possible to implement this using the SOAP API, lucky for us Salesforce has provided an ANT deployment tool called the Force.com Migration Tool that we can leverage.

Ingredients List

Here is a quick rundown of the different components that will be used for this tutorial. Some items can probably be switch out rather easily, others might be more complicated.

• Salesforce - DEV Instance + Developer Sandboxes
• GitHub - Salesforce Metadata versioning
• CentOS - Host server for Jenkins
• Jenkins - Continuous Integration Server
• Force.com Migration Tool - Interact with the Salesforce Metadata API via ANT/Jenkins
• python - Required by script for generating a destructiveChanges.xml file

Install Jenkins

I won't duplicate the effort here since I've already written up how to install Jenkins on CentOS as a service. We'll be assuming that it is running as user jenkins and installed in /home/jenkins/.jenkins.

Force.com Migration Tool

To download the JAR containing these libraries, first log into your Salesforce instance and go to Setup. From the site menu navigate to App Setup > Develop > Tools > Force.com Migration Tool. Next we need to put this JAR into a location where ANT can find it - the ~/.ant/lib/ant-salesforce.jar is probably the best option. Note that if you are running Jenkins under a different user, you will need to navigate to that user's home folder.

Create Jenkins Project

Once you've started and logged in to Jenkins, you first need to make sure you have all the requisite plugins. From the dashboard... need to create a project to use to do your builds. To do this, from the homepage...

• Click "Add Item"
  • Enter a project name (the examples below assume the project/folder name is "project")
  • Choose "Build a free-style software project" as the project type.

Make sure to specify "github" as the local subdirectory for the git checkout - or another directory of your choosing as long as it is correctly specified in build.properties and the destructiveChanges.py -d and -g parameters.

ANT Configuration: build.properties

Here is where you set your username, password (and api token), target url, and build subdirectories.

# build.properties
# http://www.salesforce.com/us/developer/docs/daas/index.htm

# Specify the login credentials for the desired Salesforce organization
sf.username = user@example.com.environment
sf.password = MyPassword123!yALBudEasdf23DsaStewCZbI7
#sf.pkgName = <Insert comma separated package names to be retrieved>
#sf.zipFile = <Insert path of the zipfile to be retrieved>
#sf.metadataType = <Insert metadata type name for which listMetadata or bulkRetrieve operations are to be performed>

# Use 'https://login.salesforce.com' for production or developer edition (the default if not specified).
# Use 'https://test.salesforce.com for sandbox.
sf.serverurl = https://test.salesforce.com

sf.maxPoll = 100
# If your network requires an HTTP proxy, see http://ant.apache.org/manual/proxy.html for configuration.

# Subdirectories, used in build.xml and destructiveChanges.py
sf.destructiveChanges = ./destructiveChanges
git.dir = ./github/src

ANT Build file: build.xml

Here is where we call the Salesforce Metadata ANT targets. Notice that we use the package.xml from the Git repository.

<project name="Salesforce Ant tasks" default="compile" basedir="." xmlns:sf="antlib:com.salesforce">
  <property file="build.properties"/>
  <property environment="env"/>

  <!-- Compile the contents of our GitHub repository -->
  <target name="compile">
    <sf:deploy username="${sf.username}" password="${sf.password}" serverurl="${sf.serverurl}" maxPoll="${sf.maxPoll}" deployRoot="${git.dir}" runAllTests="false" />
  </target>

  <!-- Get the contents of our Saleforce environment based on the package.xml from GitHub -->
  <target name="retrieveUnpackaged">
    <!-- Make sure the environment is clean -->
    <delete dir="unpackaged" failonerror="false"/>
    <mkdir dir="unpackaged"/>
    <!-- Retrieve the contents into another directory -->
    <sf:retrieve username="${sf.username}" password="${sf.password}" serverurl="${sf.serverurl}" maxPoll="${sf.maxPoll}" retrieveTarget="unpackaged" unpackaged="${git.dir}/package.xml"/>
  </target>

  <!-- Remove code in the destructiveChanges subdirectory, populated by destructiveChanges.py -->
  <target name="undeployCode">
    <sf:deploy username="${sf.username}" password="${sf.password}" serverurl="${sf.serverurl}" maxPoll="${sf.maxPoll}" deployRoot="${sf.destructiveChanges}"/>
  </target>
</project>

Salesforce Metadata Destructive Changes

Since changes are usually additive we need to parse out the things we want to remove. This gets a little crazy with some long regular expressions and callouts to a shell, but it's easier to do what we need leveraging diff, grep, and sed than python. All together it will package the metadata items to remove into a destructiveChanges.xml file so that it can be removed from your target instance. You can pass in -d to specify a directory other than "destructiveChanges", -f to use a different filename than "describe.json", and/or -g to specify a git checkout subdirectory other than "github".

#!/usr/bin/env python
import getopt, json, os, sys, StringIO

def main(argv):

	describe_filename, base_dir, git_dir = 'describe.json', 'destructiveChanges', 'github'
	try:
			opts, args = getopt.getopt(argv,"hf:d:g:",["file=","dir=","git="])
	except getopt.GetoptError:
			print os.path.basename(__file__) + ' -i <inputfile> -o <outputfile>'
			sys.exit(2)
	for opt, arg in opts:
		if opt == '-h':
			print os.path.basename(__file__) + ' -f <describe.json> -d <destructiveChanges> -g <github>'
			sys.exit()
		elif opt in ("-f", "--file"):
			describe_filename = arg
		elif opt in ("-d", "--dir"):
			base_dir = arg
		elif opt in ("-g", "--git"):
			git_dir = arg

	describe_file = open(describe_filename, 'r')
	describe = json.load(describe_file)
	describe_file.close()

	if not os.path.exists(base_dir):
		# create the directory
		os.mkdir(base_dir)
		# add an empty packages.xml
		xml = '<?xml version="1.0" encoding="UTF-8"?>\n'
		xml +='<Package xmlns="http://soap.sforce.com/2006/04/metadata">\n'
		xml +='	<version>28.0</version>\n'
		xml +='</Package>'
		package = open(base_dir+'/package.xml', 'w')
		package.write(xml)
		package.close()

	if os.path.exists(base_dir+'/destructiveChanges.xml'):
		os.remove(base_dir+'/destructiveChanges.xml')

	proc = os.popen("diff -rq "+git_dir+"/src/ unpackaged/ | grep 'Only in unpackaged' | grep -v 'meta' | sed 's/Only in unpackaged\/\([^:]*\): \([^\.]*\)\.\(.*\)/\\1,\\3,\\2/g'")
	diffs, metas = proc.read(), []
	buf = StringIO.StringIO(diffs)
	for line in buf.readlines():
		line = line.strip()
		data = line.split(",")
		if data[0] in describe:
			meta = {
				'member': data[2],
				'name':''
			}
			if data[1] in describe[data[0]]:
				meta['name'] = describe[data[0]][data[1]]
			elif '*' in describe[data[0]]:
				meta['name'] = describe[data[0]]['*']
			else:
				continue
			metas.append(meta)

	customfields = []
	proc = os.popen("diff -rqw "+git_dir+"/src/ unpackaged/ | grep 'object differ$' | sed 's/Files \\(.*\\) and \\(.*\\) differ/\\1,\\2/g'")
	diffs, objs = proc.read(), []
	buf = StringIO.StringIO(diffs)
	for line in buf.readlines():
		line = line.strip()
		data = line.split(",")
		git_file, sfdc_file = data[0], data[1]
		class_name = git_file.split("/")[-1].replace('.object', '')
		proc = os.popen("diff "+git_file+" "+sfdc_file+" | grep fullName | grep -v '^<' | sed 's/.*fullName>\(.*\\)<.*/\\1/g'")
		diffs = proc.read()
		buf2 = StringIO.StringIO(diffs);
		for line2 in buf2.readlines():
			line2 = line2.strip()
			customfields.append(class_name+'.'+line2)	

	if len(metas) > 0 or len(customfields) > 0:
		xml = '<?xml version="1.0" encoding="UTF-8"?>\n'
		xml +='<Package xmlns="http://soap.sforce.com/2006/04/metadata">\n'
			if len(metas) > 0:
				for meta in metas:
					xml +=' <types>\n'
					xml +='		 <members>'+meta['member']+'</members>\n'
					xml +='		 <name>'+meta['name']+'</name>\n'
					xml +=' </types>\n'
			if len(customfields) > 0:
				for field in customfields:
					xml +=' <types>\n'
					xml +='		 <members>'+field+'</members>\n'
					xml +='		 <name>'+field+'</name>\n'
					xml +=' </types>\n'
		xml +='</Package>'
		destructiveChanges = open(base_dir+'/destructiveChanges.xml', 'w')
		destructiveChanges.write(xml)
		destructiveChanges.close()
		print "CREATED: destructiveChanges.xml"
		print xml

if  __name__ == '__main__':
	main(sys.argv[1:])

Salesforce Metadata describe.json

A JSON formatted list of metadata types supported by your org. Maps the filename to a particular type.

{
    "labels": {
        "labels" : "CustomLabels"
    },
    "staticresources": {
        "resource" : "StaticResource"
    },
    "scontrols": {
        "scf" : "Scontrol"
    },
    "components": {
        "component" : "ApexComponent"
    },
    "pages": {
        "page" : "ApexPage"
    },
    "queues": {
        "queue" : "Queue"
    },
    "objects": {
        "object" : "CustomObject"
    },
    "reportTypes": {
        "reportType" : "ReportType"
    },
    "reports": {
        "report" : "Report"
    },
    "dashboards": {
        "dashboard" : "Dashboard"
    },
    "analyticSnapshots": {
        "snapshot" : "AnalyticSnapshot"
    },
    "layouts": {
        "layout" : "Layout"
    },
    "documents": {
        "*" : "Document"
    },
    "weblinks": {
        "weblink" : "CustomPageWebLink"
    },
    "tabs": {
        "tab" : "CustomTab"
    },
    "customApplicationComponents": {
        "customApplicationComponent" : "CustomApplicationComponent"
    },
    "applications": {
        "app" : "CustomApplication"
    },
    "letterhead": {
        "letter" : "Letterhead"
    },
    "email": {
        "email" : "EmailTemplate"
    },
    "workflows": {
        "workflow" : "Workflow"
    },
    "assignmentRules": {
        "assignmentRules" : "AssignmentRules"
    },
    "autoResponseRules": {
        "autoResponseRules" : "AutoResponseRules"
    },
    "escalationRules": {
        "escalationRules" : "EscalationRules"
    },
    "roles": {
        "role" : "Role"
    },
    "groups": {
        "group" : "Group"
    },
    "homePageComponents": {
        "homePageComponent" : "HomePageComponent"
    },
    "homePageLayouts": {
        "homePageLayout" : "HomePageLayout"
    },
    "objectTranslations": {
        "objectTranslation" : "CustomObjectTranslation"
    },
    "flows": {
        "flow" : "Flow"
    },
    "classes": {
        "cls" : "ApexClass"
    },
    "triggers": {
        "trigger" : "ApexTrigger"
    },
    "profiles": {
        "profile" : "Profile"
    },
    "permissionsets": {
        "permissionset" : "PermissionSet"
    },
    "datacategorygroups": {
        "datacategorygroup" : "DataCategoryGroup"
    },
    "remoteSiteSettings": {
        "remoteSite" : "RemoteSiteSetting"
    },
    "authproviders": {
        "authprovider" : "AuthProvider"
    },
    "sites": {
        "site" : "CustomSite"
    },
    "leadSharingRules": {
        "sharingRules" : "LeadSharingRules"
    },
    "campaignSharingRules": {
        "sharingRules" : "CampaignSharingRules"
    },
    "caseSharingRules": {
        "sharingRules" : "CaseSharingRules"
    },
    "contactSharingRules": {
        "sharingRules" : "ContactSharingRules"
    },
    "opportunitySharingRules": {
        "sharingRules" : "OpportunitySharingRules"
    },
    "accountSharingRules": {
        "sharingRules" : "AccountSharingRules"
    },
    "customObjectSharingRules": {
        "sharingRules" : "CustomObjectSharingRules"
    },
    "communities": {
        "community" : "Community"
    },
    "callCenters": {
        "callCenter" : "CallCenter"
    },
    "settings": {
        "settings" : "Settings"
    }
}