VRize is a startup which provides end-to-end virtual reality enterprise solutions.They are launching VRize Video which is a CMS (Contents Management Service) for help a customer to provide multi-device VR video application.

On this service, once a customer submits their application on the VRize video, then the VRize video automatically kicks Jenkins and build the application for multi-platform. Then it uploads the apps to the blob storage.
The customer gets notification finishing the build. Finally, the customer can download new apps for a lot of platforms.

However, they need to solve several problems to improve their system to get more clients. They decided to have a two-days hackfest with Microsoft technical evangelist to fix this problem.

Core Team:

Hitoshi Tsuyuki – VRize
Mizuki Kameda – VRize
Tsuyoshi Ushio – Technical Evangelist, DevOps, Microsoft
Daisuke Masubuchi – Technical Evangelist, Startups, Microsoft

Problem Statement:

They already develop the service and pipeline. The biggest pain-point is build time. Since they need to build apps for multi-HMD (Head Mounted Device) like Oculus、PlayStationVR、HTC VIVE、GearVR、Daydream. Also, they build for iOS, Android,
Windows, and PS4.

That is why it takes some time to build apps for a lot of HMD/Platforms. They need to build a lot. Currently, we can estimate
the build time like this. Build time is 2-15 minutes for one platform. The platform switch takes about 10 minutes to change the target platform. It just switches the platform. Since the platform switch takes time, they want to avoid the platform switch.
Also, they need whole repeat operation for every HMDs. If it takes a long time, the customer will get frustrated.

The second issue is the automation. For building HMD application, it requires Unity. Unity is a great tool. However, it
is very hard to automate. We can automate some part. But you need to activate the license manually. Also, installing
Unity takes a lot of times about one hour. If we autoscale their build system, it would take a long time to bootstrap their build server. That is why they wanted to know how to automate the VR build a pipeline.

Solution Overview

We solve this problem using OSS based technology on Azure. They are used to using OSS based tool. We use Packer
For building a VHD image for Jenkins slave. We already install Unity and some tools for automation to the VHD image.
Using the VHD image, we provision VMScale Set, which enable us to autoscale, with the VHD image. We create VM Scale set for the platforms. (e.g. VMSS for iOS, VMSS for Android … and so on.)

When the VM Scale set instance boot ups, the custom script extention executes a script for platform switch for every pipeline. A VMScale set instance executes it only once.

Using this architecture, if you can’t fully automate the manual operation related windows-only-operation, you can operate it only once after the packer
creates the VHD image. Then you can use the new VHD image the provision for Jenkins slave is fully automated and use it for autoscaling.

Let’s get hack it!

1. Provisioning k8s Cluster

They want to use ACS(Kubernetes) of their CMS portal for their customers. They’d like to use it for several reasons.

• Scalability
• Portability of infrastructure among the cloud platform
• Availability without infrastructure engineer.

We started with provisioning and a k8s cluster.

You can create a k8s cluster by Azure Portal. However, you have alternatives. Using acs-engine, which enable us to generate ARM template for ACS cluster and also help us to customize it. You can see the Kubernetes workthorough and follow the instruction; you can easily to install k8s cluster and set up OMS.

This part is nothing special. VRize guys said that they could handle it. So we decided to focus VR build pipeline hack. The hack fest is only two days!

2. Architecture

We discuss the VR build pipeline architecture.

2.1. Provisioning Build Pipeline Servers

1. Building VHD image for VMSS

We have several Virtual Machine Scale Set for build apps for each platform. We can Scale In/Out or Autoscale using VM Scale set. It requires OS image. Instead of writing code for automating provisioning
an instance of a Virtual Machine, we wrote a script to build the OS image. We use Packer for this purpose.

We can easily use Packer for creating VHD image. However, we have some tips for creating Windows Server VHD images on Azure. We just write provisioner.json and provision.ps1 .

However, we have some tips for using Packer for Windows Server on Azure.

You can see the source code which we built on the GitHub.

• provision.json
• provision.ps1

Tips: Packer: Object ID for Windows Image

Packer doesn’t work when we don’t specify the Object ID for Windows images.

The provision.json is a template file for creating a Windows VHD image.

 {
 "variables": {
 "subscription_id": "<YOUR SUBSCRIPTION ID>",
 "object_id": "<YOUR OBJECT ID>",
 "client_secret": "<YOUR CLIENT SECRET>",
 "client_id": "<YOUR CLIENT ID>",
 "tenant_id": "<YOUR TENANT ID>",
 "resource_group_name": "<YOUR RESOURCE GROUP NAME>",
 "storage_account": "<YOUR STORAGE ACCOUNT NAME>"
 },
 :

object_id is an object id of a service principle. You can get it like this.

 azure ad sp show -n CLIENT_ID

Also, I sent an issue and pull request for contributing Hashicorp/Packer. The contributor merged the pull request after the hackfest.

Also, you need to add some Windows Related parts.

 "communicator": "winrm",
 "winrm_use_ssl": "true",
 "winrm_insecure": "true",
 "winrm_timeout": "3m",
 "winrm_username": "packer"

Finally, you need to execute sysprep for clean up user account from current image.

 "provisioners": [
 {
 "type": "powershell",
 "script": "provision.ps1"
 },

{
 "type": "windows-shell",
 "inline": ["%windir%\system32\sysprep\sysprep.exe /generalize /oobe /shutdown"]
 }
 ]

Sysprep is only for the cmd.
Then let’s see powershell file. We use chocolatey, which is the package manager for Windows. It installs chocolatey, Unity, git, python3, jdk8, and pywinauto into the VHD image.
We don’t need to install Jenkins as a slave. Jenkins master handles this.
Unfortunately, we are running out of time. We can’t have enough time for automating the installation of Android SDK. However, we can hack and remove a lot of obstacles. We’ll have the second hackfest; then we will solve this.

Now we can build a custom VHD image by this command.

 packer build provision.json

You can see the VHD file on your storage account on which you specified at the provision.json.

Also, we have some tips for installing Unity. Unity installation command is executed asynchronously, and you have no way to detect the installation end. You just need to wait for while about 10 min.

2. Provisioning Jenkins Slave

At the same time, we hacked how to provision VM Scale Set with Managed Disk using Terraform.

However, At the hackfest, someone sent pullrequest and waited for the review. NOTE: It has been merged after the hackfest.

Since it is just two days hackfest, instead of sending a pull request to the terraform, we try ARM template first.

See the repo. We also share Terraform file for the future reference.
I may update the repo when the VMSS + ManagedDisk Released.

Instead, You can see the ARM template for your reference.

• ARM Template

3. Custom Script Extension for executing platform switch

We need to separate the pipeline for iOS, Android, and so on. We need to fire platform switch command for each platform, and it takes time.

We decide to use Custom Script Extension for this purpose. The extension will be executed only the first boot time of VMSS instances.

• CustomScriptExtension

4. Automated Unity activation/deactivation

We need to activate Unity for each VMSS instance when it is bootstrap first time. We can use CustomScriptExtension for this purpose, as well.

You can activate Unity with this script

"C:Program FilesUnityEditorUnity.exe" -quit -batchmode -serial XX-XX-XXX-XXXX-XXXX-XXXX -username 'xxxxxx' -password 'xxxxxxxx' -logfile

Also, you can retrun license using this script. This feature is best fit for Scale in scenario.

"C:Program FilesUnityEditorUnity.exe" -quit -batchmode -returnlicense -logfile

However, return license doesn’t work properly in CLI. It works well when we use GUI. We are reporting this issue to Unity.

Conclusion

We just hacked just two days for the VR build pipeline. However, we have a lot of learning from this. As this PoC, we prove that this architecture works. I’ll summary the learning. We’ll have the second hackfest to implement to the production and solve the ToDo tasks for that.

We convince this architecture works and reduce the build time for their customer.

This strategy works:

• Packer for automating installing Unity and related product then create VHD.
• Provisioning VMSS with Managed disk using ARM template
• Using Custom Script Extention to activate license and platform switch when it is bootstrap at the first time.

Overall, this architecture works!

The challenges which remains:

• Automated installation of Unity Android SDK. We can’t set a path for it. (This time, time’s up for investigation for two)
• Deploy VMSS using Terraform (Waiting for the pull request merged)
• Return license of Unity deactivation

We leave some challenges; however, this architecture is fine. Now We might automate VR application build pipeline for several platform/HMD with concurrently with auto scale manner! VRize is changing the world!

I can’t wait for the next hackfest!

Customer Comments:

I really appreciate that Microsoft technical evangelists help us to solve the technical challenges, which we thought it must be very difficult, through this hackfest.I was really enjoyed two days deep technical discussion and hacking. Please have the second hackfest! – Mizuki Kameda (VRize)

Not only the advanced usage of Azure but also we solved some problems. We learned that how to create Windows VHD on Azure, how can we restore it, how to configure Kubernetes cluster on Azure.
We had some issues which are to scale out the CI/Build server, develop a DevOps environment.

We learned and solved this problem by this two-days hackfest. I appreciate Microsoft. Let’s do the second part together! – Hitoshi Tsuyuki (VRize)

Resources

• VRize Video
• Virtual Machine Scale Set
• Packer
• Packer ObjectID related pull request
• Terraform
• chocolatey
• Azure Resrouce Manager
• CustomScriptExtension
• Kubernetes workthorough
• acs-engine
• Code repo of this Hackfest