Lessons From Humana’s Migration to HCP Terraform Cloud
BOSTON — Yvette Villanueva had been on her job as Humana’s associate director of automation engineering and advancement a month when her boss decided to migrate off Terraform Enterprise to Terraform Cloud — and put her in charge of the shift.
“My tenants didn’t even know who I was yet, right? And here I am planning this huge migration,” Villanueva told audiences at October’s HashiConf in Boston. “So I thought, well, do an introduction. Hi. My name is; I’m new here, by the way. Making plans. Don’t have any dates yet, and don’t know what it’s going to take, but at some stage in the near future, we will be migrating… .”
She highlighted the benefits, of course, such as opportunities for optimization and deleting old workspaces that no one was using anyway. “There was a great opportunity and optimization there, because I knew we were moving and I was willing to have yard sales for sure, anything else to get rid of the things that we didn’t want to take with us,” she said.
Humana’s Terraform Footprint
Humana is not a small organization. The healthcare insurer has more than 16 million members and 67,000 employees across the U.S. and Puerto Rico. Terraform managed 2695 workspaces across five environments. V
Villanueva’s team is responsible for core enterprise engineering services, and specifically, it is an automation platform, responsible for Terraform, Ansible and a few other solutions.
First, she knew she needed more data, which she was able to get because she had a PowerShell master on the team. The data showed a breakdown of the 2695 workspaces by the five environments, called A through E.
“It also showed us that we had almost 300 approved modules available for users,” she said. “We didn’t have visibility to the utilization of the modules, so I knew I had 300 modules that could potentially be utilized at any time.”
It also showed there was room for optimization.
“The footprint of the component to workspace utilization could be simplified, and we could then have a smaller footprint in workspaces for particular teams and particular organizations,” she said. “So what we knew is we could reduce our workspace footprint prior to moving.”
Planning the Migration
The team started by “building the runway,” she said. A runway is the process of setting up the necessary infrastructure and environment for a smooth and successful migration.
Humana’s Runway for the migration off premise to the cloud. Photo by Loraine Lawson
“A big thing for me was simplifying the migration and requiring a minimal level of effort from our tenants and from all of our dependencies, so networking and everything else,” she said. “When we talked about standing up our HCP TerraForm agents, we decided that we would use the same namespace as our existing TerraForm enterprise agents.”
Her job might have been simplified by Terraform Migrate, which was released at the conference, but obviously, it wasn’t available at the time. There was another existing option, but it would have required Humana to migrate all its workspaces at once, an impossibility given Humana’s 2600 workspaces across 132 organizations, she added.
The team had to build its own automation, which it did using some Terraform modules that the company took apart and reused with their own hybrid solution written in Powershell.
“We built automation to create the new workspaces to copy the variables over, build the team relationships, ownerships and move state files,” she said.
The team had two automation workflows it used to build new workspaces. They had to pivot those workflows to start building HCP TerraForm workspaces instead of Terraform Enterprise workspaces.
“One of those workflows we owned and we managed. That was ours. We could do as we wanted, and so that was going to be easier than our second,” she said. “Our second workflow was tightly coupled with another platform/environment owner, and we were going to need to work with them to build that automation and make that pivot happen.”
The third piece of the workflow shift was the Terraform modules. They had approximately 300 Terraform modules that needed to be refactored for HCP Terraform. A sibling team, service enablement, was responsible for that so her automation team partnered with them to ensure they were working within the migration timelines and supporting the effort, she said.
Simplifying the Journey
“Obviously, there were things that were variables and things that are related to security that we would not be able to move for them, and we had different iterations also of maturity, in terms of our deployments and pieces like that,” she said. “We definitely needed to identify what we could do for them.”
Humana’s simplified migration path for different teams. Photo by Loraine Lawson
Her team began to work with the handful of teams with a huge footprint.
“I didn’t have to do a lot of convincing, even though there was a level of effort on their part,” she said. “What I did need to do is provide an engineer that had the vision and could guide them to what we were recommending so they can make that happen.”
Her team also built some automation that could help the process by cleaning out unused workspaces based on criteria the team established as a policy. That was communicated to the other teams.
“The intent here was really to highlight that it will be simple. It’s a big ask in terms of number of workspaces, but in terms of level of effort to our tenants, it can be mitigated and controlled,” she said.
The tenants of environments A through E could be moved independently of one another, whenever they were ready. The team started by standing up Terraform, copying over their workspace. The other teams could then handle moving their variables and doing their validation. After that, she would be able to sunset Terraform enterprise. However, the platform team had an environment that required all of the workspaces to be moved at the same time.
“This is also the environment that had the workflow that was creating their own enterprise workspaces, and they would have to build automation to adjust that,” she said. “So after months of working our simplification and optimization, getting our runway built — so infrastructure, all those good pieces, some of our governance processes — we were ready to actually welcome people into HCP Terraform.”
At that point, they had number of workspaces and two migration paths.
Build It and They Will Procrastinate
To help with the migration, Villanueva also set up rapid migration sessions. The sessions were set up so that on Wednesdays and Fridays, the teams could come in and work with an engineer who could answer questions and help. She’d had success with this approach before in her career; it seemed like a good plan.
“HCP TerraForm is open for business. We’re excited. A communication goes out to all the teams, notifying them you can request at any time now to be migrated over,” she said. “We execute the automation to actually create the workspaces and move their variables and all that, and we’re waiting for them to come tell us, just move my state file. I’m ready.”
What happened, though, is that a few teams showed up, resulting in only 1% of workspaces migrated after her announcement.
“It was disheartening, but from the conversations that I was having with teams, either the ones we were working to optimize with, or just teams within my blast radius, as I like to say, one thing that was the common denominator is that they had competing priorities,” she told audiences. “I really was offering them the door was open, but I wasn’t telling them when I was closing the other door.”
A graph showing the Humana to cloud migration timeline. Photo by Loraine Lawson
It was time for a pivot, and so the migration team decided to handle the migration for them. But that was tricky, because the other teams still needed to bring in their sensitive variables and other things to which the migration team did not have access.
“We also knew we needed to build automation to lock their TFE workspaces, right? So we scheduled a migration for them. We built the automation to lock those Terraform workspaces for environments A through D, we communicated to the teams that their TFE workspaces were not going to be available after that specified date, and we gave them time, and then we executed the migration on the specified date,” she said.
The migration team ran the automation scripts, created the new workspaces, variables, all the relationships, moved their state files and then locked their enterprise workspaces. But some weren’t ready to go.
“They had competing priorities, and their environment is a significant environment in our roadmap at Humana,” she said. “There was a lot of working to ensure that we were going to get the cycles that we needed from the team members and at what point.”
The migration team worked with those teams to determine when they would make the shift, which led to some scary times for Villanueva.
“This little green star represents the time when environment and platform owners told us that they actually weren’t going to be able to migrate and do everything that we were asking of them until April of 2024, so 12 months is what they were asking for about here. This is when I started stressing a little bit about the license count, maybe just a little bit,” she said.
Lessons Learned
“As you guys can see, the moving of the workspaces was a much smaller piece, especially when you move them for them, my roller coaster ride,” she said.
The majority of the team’s time was split between communication/collaboration, maintenance, optimization and planning, with planning including tasks such as contract negotiations, strategy, data extraction, visualizations, and analysis.
Lessons learned from Humana’s migration from Enterprise Terraform to HCP Terraform, a cloud platform. Photo by Loraine Lawson
Governance can seem like a headache but it was a vital part of the initial migration preparation, according to Villanueva. While she spent “a sliver of time in the internal governance board,” that time simplified the migration path by producing artifacts for it. It made it easier because teams didn’t have to think about security or other policies related to the migration.
She had joked about having a yard sale for unused workspaces before the migration, but her strategy of purging unused workspaces from the infrastructure paid off by simplifying the process.
“The maintenance and optimization helped us to recoup workspaces,” she said.
She also learned a hard lesson when only one percent moved despite her communications push. It taught her that dates help with competing priorities.
“What this taught me was that my excitement doesn’t translate to everybody else because they’re busy, right?” Villanueva said. “Some people were excited and they were the first ones came knocking on the door. They were waiting for us to be able to give them those workspaces, but then everybody else, they were busy.”
She also found was important to hold onto workspaces rather than just delete them immediately. That provided users with a sense of safety.
“What we had told users is we’ll hold on to them for 60 days, and that way you could feel warm and fuzzy that no matter what, it’s sitting there, and so that was a big part of us having to strip out those permissions because those workspaces were going to exist for 60 days,” she said.
Finally, plan, do, check and act — but be ready to adjust your plans, she recommended.
“A few times in this migration, as we were looking at some of the reporting that we built to track state files and update dates and who’s using what and where, we just realized that the migration wasn’t moving the way that we wanted to, and it gave us an opportunity to adjust our plans,” she said.
HashiCorp paid the travel expenses for the reporter to attend this conference.
