I spent a little more than ten years as a client before I became a consultant. I spent just a bit longer as a consultant before becoming a client again. There are things I’ve learned in each role that help me do the other better. To ensure a mutually beneficial engagement it is helpful to understand the perspective of the other team.

Why is understand both sides useful?

The goals of a consultant and a client organization are misaligned. That doesn’t mean you can’t do great things together, but if you don’t understand the goals of your partner you are likely to step on each other’s toes.

The goals of clients

Client looks to consultants for one of two primary reasons:

• Add to team capacity
• Solve a problem

We either need to complete a project that our team does not have the time to tackle, or we need expertise it does not make sense for us to keep on staff. Sometimes we’re looking to reduce costs by having a group of part time people fill the roles of a smaller number of fulltime team members.

I like to have my team’s staffing level sufficient to complete all day-to-day tasks and to bring in outside help to take on special projects. Other people like to have consultants around consistently to provide the outside perspective and the diverse expertise that consultants bring. Both of those strategies are foundationally aimed at those two needs.

A smart client wants to spend the money needed to be successful, but not more. We want the most value for our money we can possibly get.

The goals of consultants

Consultants have a different pair of primary goals:

• Record billable hours and/or ensure profit margin
• Have a happy client who refers more business

Some consultants will protest that they have the goal to solves client’s problems through good work. My perspective is those are way to achieve those two goals. Some clients are happy when you do good work (but not all). All clients are paying to have a problem solved (see above).

Profit motive isn’t evil or wrong – even when supporting nonprofits and other socially beneficial institutions (having spent much of my career in nonprofits, we think about this a lot). Consultants need to make money to stay afloat. A consulting firm has people to pay, overhead to manage, and founders/investors to reward. Independent consultants need to eat, pay their mortgage, and so on. The larger the firm, the more pressure there is for larger profit margins.

To get new clients, consultants need “referable” clients. That means having clients who are so happy with the work done they will serve as a reference. I wish that always meant creating the best solution possible. What it means in practice is building the solution that makes the client happy. As a consultant I gave clients my best advice, and when they disagreed and insisted on a different solution, we build that instead. If they ran out of money along the way we still tried to keep them happy, even if we had the duct tape the last bits.

In the end, consultants build what clients pay for, and that’s not always the best solution.

Finding the balance

With consultants trying to make the most money they can, and clients trying to get a successful solution for the least money, there is an inherent tension in the system. Still, there is a balance to be had, where everyone wins, and great things happen. The trick is to make it a healthy tension that forces everyone to be better. Finding that balance doesn’t require that everyone involved has spent time on the other side the relationships, but it certainly helps.

When you understand the needs and goals of the other side of the relationship you can adjust your approach to make sure everyone is aligned to win.

Lessons to take from being a client

One of the things I learned along the way was that a lot of the advice given to new consultants contradicts what I knew from being a client. Spending time as a client gives you insights into how to best serve customers that many pure consultants don’t understand.

Be the consultant you want to hire

When you work at an organization that hires consultants you see different approaches taken by different firms. You learn your preferences about what you like and don’t like in a consulting partner. While no one style is the best fit for everyone it’s unlikely that you are so unique that there aren’t lots of other people who like that same style.

Default to the Golden Rule: treat clients the way you wanted to be treated by consultants.

You can’t always do that 100% of the way – sure as a client I want everything free, but that’s not reasonable. But by approaching the client the way I would have wanted to be treated consistently went a long way to helping smooth over challenges.

Start there, and over time you’ll learn to adapt your approach when specific clients prefer a different style.

Be honest about your limitations

Do. Not. Lie. To. Me.

Do not guess without admitting it. If I wanted made up answers, I’d ask an AI.

Consultants always want to appear to be the expert in the room, and so they feel they have to answer every question. Too often that leads to consultants making up answers to show how smart they are; clients will catch you eventually.

One of the best ways to build trust with a client is admit when you don’t know the answer to a question, and then come back later with the answer. Do not say “I don’t know” and leave it there, go for some form of “I will need to go look that up/ask around/figure that out.”

Great consultants find solutions, they don’t always have the answer right away. We can wait for you to do some research when we stump you. That is a lot easier to explain than when you have to walk back having given us the wrong answer.

Focus on what the client needs to succeed

Clients should always have an outcome in mind that supports their work. Consultants are focused on the solution they are building. When everything is going well, that solution is what the client needs to support their work. If those stop being the same thing you have a very big problem.

Both clients and consultants can easily forget to consistently re-check that alignment. As a client and as a consultant I’ve been part of projects where the delivered solution didn’t solve the actual problem – even when it fulfilled the spec and SOW. These moments frequently lead to energetic discussions that often become loud. No one wins when that happens.

Regularly check with the client, and with yourself, to see if the solution will solve the client’s problem. When you see misalignment raise your hand early and often.

Lessons to take from being a consultant

Of course consultants know and learn stuff that isn’t obvious to any given client. Consultants bring wider experiences, different perspectives, and a different energy to a project. That is part of what makes them valuable. Clients should hire a consultant they trust, and listen to their consultant. Think hard before deciding you know better.

Always learn new things, even if they aren’t important today

As a client we tend to learn deeply about the tools we use and our work. Consultants work on a lot of projects with a lot of clients. Along the way they use a lot tools, and see at lot of ideas. That creates a culture and need for constantly learning. Often they are learning about things that don’t seem useful right away.

The higher the role you have as a consultant, the more you are expected to be at least conversant about technology you haven’t used yet. You also need to be conversant about the work of your clients. That’s a lot of learning.

I had good learning habits going in to being a consultant. They served me extremely well as a consultant, and are serving me well again as a client.

The broad knowledge of a consultant is extremely useful and everyone benefits from more people knowing more stuff. Having that breadth of knowledge also helps when you do run into the places where you don’t know something. It gives you the confidence that you can go learn the next thing you need to know quickly (see Be honest about your limitations above).

Know how to work to a deadline

Consultants are always working within time and budget constraints – usually tight ones. That forces them to learn to be efficient. Sometimes that means they cut corners (see next section) usually that just means they move fast. Good consultants have a high degree of dexterity with their tools, they learn to line up their work to knock out tasks, and they learn what’s needed and what’s just nice to have.

New consultants often feel like they are sprinting all the time, but experienced consultants learn to balance the sprints with jogging. The pace is nearly always high (at least if sales are going well), but it still ebbs and flows. Consultants learn to hit their deadlines, but rarely are ready to deliver early.

As a consultant if a deadline was far in the future it gave me time to do careful work, balance other clients, do research, or just time off. Far off deadlines gave me time to recover from sprints and make sure I had the energy for high intensity moment. That intensity is important to driving client success – but hitting the deadline is more important.

Hitting deadlines is also important for a client to do. Consultants need you to hit your deadlines so they can balance their workload to hit their deadlines. They may also have penalties embedded in the contract (see Read the Contract below) that could cost you time or money over the course of your project.

Perfect is the enemy of the good

Okay, this isn’t something just consultants know, but it is something consultants often learn to deal with the hard way.

Consultants need a solution that meets the requirements, fits in the budget, and pleases the client. They are not there to create a solution that is perfect, or even elegant. In any project there is a balance to be had between carefully polished, and just barely good enough to be successful. Consultants learn to thread that needle. As long as the project is successful that’s a good thing.

I have seen developers spend hours, days, even months, trying to build to the perfect level of abstraction, with the perfect naming conventions, and drive for the perfect code, only to have the project fail because it’s overdue, over budget, and was outmoded by someone who worked twice as fast.

Yes, we all want good solutions to our technical problems. But no solution is going to be perfect. You should aim for perfection and know you are going to miss. When you learn to accept that, it’ll be easier to move forward and be successful.

Things everyone should know regardless of role

For all there are things that each side brings something to the table, there are habits that everyone should have as part of their role. There are lessons I learned, or was taught, in both roles that are super important.

Read, and understand, the contract

Everyone on a project benefits from having working knowledge of the contract. In the end, when push comes to shove, all that matters is the words on the paper. You can usually avoid the pushing and shoving by understanding what everyone agreed to up front.

The biggest issues I’ve seen on consulting projects was when one side, or the other, didn’t pay attention to the agreement.

Sometimes this happens because everyone is working in good faith, and no one remembers to amend the agreement when needs changed. In those cases you can often recover by continuing to work with each other in good faith.

Sometimes this happens when someone signed a contract they didn’t read and understand. I once had a client yell at me because I added a paragraph to the contract outlining the resources they were responsible for providing and he didn’t read it before we asked him for those resources (these clauses are really standard, and the one I wrote was extremely simple).

If everyone on the team takes the time to read and understand the contract it greatly reduces friction. Clients who understand the bounds and assumptions in a contract are able to get the most from their vendor without creating tension. Consultants who track the required deliverables of the contract don’t frustrate clients by skipping required elements. It doesn’t take long. The more you read them the faster we’ll be at reading the next one.

Once you have read a bunch of contracts you’ll know what’s normal and what’s not. At this point, if I don’t understand the contract language I see that as a red flag even before I send it out for legal review.

Discuss problems and be solution oriented

Projects go best when everyone is open about what problems exist and then pivots to solving them.

Technology should be deployed to solve problems. That means starting by talking about problems. Being problem focused at the start makes it easy to be hung up talking only about those problems, or about new problems that come up while solving the first problem.

Having a good problem statement is critical to creating good solutions. But once you have the problem outlined you need to focus on solving it. Yes, raise problems, concerns, challenges, threats, weaknesses, etc. Talk openly about all those things. Then make the pivot into problem solving mode once the issue is well understood.

The best projects come together when when everyone collaborates on finding the best solutions to the problems at hand.

Quality matters

Everyone needs to focus on the quality of the outcome. Consultants, for all their fast moving creation of imperfect solutions, must still do good work. Clients should hold their vendors, and themselves, to high standards.

Every message that goes back and forth is a chance for misunderstand that gets in the way. Every input into discovery and every deliverable is a chance for gaps to form. If anyone takes their eye off the ball mistakes can happen and the solution no longer threads the quality needle correctly.

Mistakes will happen, and everyone will have to help course correct. But the higher the quality of the work done before the mistake, the faster it will be to recover and better and overall solution the client will get.

The Grass is Greener

One final note on the way out. If you are trying to decide between being a consultant or being a client, I recommend the switch – whichever you are today try being the other if you haven’t yet. Not everyone loves both roles, and different roles have been right for me at different times.

As a client I loved what I did. We were helping make the world better. I was pushing things forward and helping the organizations succeed. But eventually the things they needed me to learn, and the pace I wanted to grow, weren’t aligned to the organization’s needs.

I’d been there a decade, I left on great terms, but it was time to go.

When I first became a consultant it was exciting. I got to work on a variety of projects, with more technologies than any one organization generally needs. The pace was higher and I was frequently pushing myself in new directions. Consulting gave me insights into how different organizations worked (for both better and worse). And I made more money.

Interesting work, exciting environment, more money, great!

As a consultant I spent less time in positions, the billable grind was exhausting, I missed being focused. When I returned to the client side, I got to focus again. I have one org to worry about, one set of organizational politics to understand, and so on. I get to learn the work of the organization deeply again and really understand the market we serve. In my case I, again, got more money – but that was at least partially luck as much as anything; consultants are often paid better than in-house team members.

Focused work, no billable hours target, calmer work environment, great!

Each really does have it’s advantages. But so does understanding what it’s like to be the person on the other side the relationship. Try them both, learn from both, decide what’s the best fit for you.

I changed jobs recently. In the process of leaving my old job I was asked a lot of questions about how I’ve built up a wide-array of skills and knowledge. During those discussions I tried to give the best advice I could on the fly to a question I was asked cold. Now I’ll try to assemble a set of more carefully considered advice.

These suggestions are more or less in the reverse direction of importance: the more you read the more valuable I think the suggestion is.

Directed Learning (Job Related Learning)

By “Directed Learning” I mean learning that is supported or encouraged by your employer. This can be certification preparation, employer provided trainings, or research into an emerging technology, even building tools that directly support your work. Into this category goes anything that can lead to higher pay, advancement, or even just keeping your current job.

Some employers are really good about this. Over time I’ve been sent to trainings, had course fees covered, and other direct support. The job I just left even provided financial incentives (cash) for us to acquire select certifications of value to the company. Other employers have published a list of what they wanted people to have – without direct incentive. But most often I’ve had to figure this out on my own which is not ideal of anyone involved.

This type of growth has a few benefits. First, it can help with your reputation (and hopefully pay) with your employer: “Oh, Aaron is certified on [interesting product] we can assign him to [high profile project].” Second, it helps make sure you are growing in alignment with your employer’s needs. That can be helpful when they are forced to make cuts: “We should keep Aaron, he’s the only team member with [rare certification] and Salesforce wants us to have that on the team.”

In my life as a Drupal developer this looked like creating modules for internal use, or learning new techniques. But it boils down to the same thing: learning stuff your employer needs you to know.

The downside of only learning what your employer needs is that you miss chances to grow beyond their needs. That limits your future mobility – your best chances will be for another job like you have today. If you want a better job or a job doing something different, you need skills your employers doesn’t think they care about.

Personal Learning: Side Projects and Other Self-Directed Research

To learn things your employer does not care about you will likely need to put in your own time. You should focus this kind of learning on something that interests you – make it fun. My experience is that this kind of learning goes best with side projects.

Side projects allow us chances to test and practice the ideas we’re learning. I use them to learn programming languages, frameworks, and techniques that interest me but not directly useful to my work. Sometimes they come around to being useful in my work. Sometimes they just give me a chance to learn interesting things. My Github account is full of examples.

An early professional mentor told me to learn a new programming language every year, and so over time I taught myself a bunch of programming languages. These days I generalize that advice a bit and try to learn a new platform, framework, or technique every year – and sometimes a new language. That meant when Salesforce rolled out LWCs to replace Aura Components, I was ready to learn them quickly. It also means when a friend started to talk about using Wails I could related from my experiences using Electron. My experience creating a project estimation tool helps me discuss estimation techniques with project teams.

This blog is an example of a side project. Every post gives me practice at writing. To keep posts accurate I include research as part of my writing process. That research teaches me things I wouldn’t learn otherwise. The act of writing forces me to clarify my thinking on topics. All of that is useful to me and helps me be better at my work. I also know this blog as been read by colleagues, employers, and potential employers all of whom are useful audiences when getting new work.

When I give talks about communication skills I tell people what they do for practice doesn’t matter: what matters is that they practice. Same is true for side projects. What project you tackle isn’t important. What is important is that you do something that forces you to learn new things.

Side projects help us look at our work from different angles, and therefore test our limits and grow.

Develop Learning Routines

It’s easy to get lazy or to put your professional development aside – it feels like there will be time later. So you should create a routine that works in your life.

I like to have a routine to make sure I’m steadily doing some form of professional development – it’s a marathon not a sprint. For me that generally looks like job-related learning for about an hour after my formal work day and working on a side project on weekends. Attaching your learning to your work day allows you to more easily block the time in your mind and with family.

I know people who like to do their study before work because that’s when they are fresh and study gets their brain into the right gear for the day. Other friends tell me they like to do research or study after their kids go to bed because it gives them more time with the most important people in their lives. All of those are great patterns because they are tailored to their life.

Whatever routine works for your life is a good one. The important part is to making learning a regular activity.

Share What You Know

One of the best ways I know to learn is to teach. Giving talks, writing blog posts, and mentoring all give me a chance to share what I know with other people.

Sharing can be intimidating. I have been writing this blog for nearly 10 years. I gave me first conference talk more than 10 years before that. And still, every time I take on a new topic I’m nervous I’ll be found to be a fraud.

I try to leverage the fear I feel to help me assemble the best content I am able (at least within my time and resource constraints). When I am writing, or preparing a talk, I also do a lot of fact checking. Since I never want to put anything into the world that I can’t back up later, I do my best to make sure I’m right. In the process I also learn more about the topic. That even means correcting information I was preparing because my initial understanding was wrong.

On the flip side I try not to let the perfect be the enemy of the good. I require myself to post on my blog once a month – limiting my editing time. Similarly conference talks have a hard deadline. Those deadlines force me to complete content on a schedule, not when it’s perfect. So be it.

When you share what you know people start to see you as an expert. That can lead them to ask you questions that push you to learn more.

All that can lead you to be an expert.

Make Friends

Making friends is easier for some of us than others. And maintaining friendships in our current society is extremely hard. But friendships are super important. They help keep us healthy, mentally fit, and happy. Friends also teach us really important things about the world around us.

If you have read this blog much you are used to seeing me make references to my friends as sources of information. They are my greatest resource for learning (and a bunch of other things too). Friends know stuff we don’t know. Even better, they will tell you things you don’t know. Good friends don’t judge you for your gaps, but are happy to help you fill them.

I value the input and perspective that friends share with me. We don’t have to agree on everything. We don’t have to be interested in the same technologies. In fact both of those conditions help me understand things I wouldn’t know about otherwise.

I cite my friends in my writing because they are important. They deserve credit when they teach me things. Processing ideas with my friends helps me test them for quality. Often they point out errors in my thinking. Those are all good things.

Be Curious

This is one of the most important lessons I learned from my mom. She was a deeply curious person and that rubbed off on my sister and I. We both remember her dragging us to lectures at the RPI Freshwater Institute in Bolton’s Landing, NY (it’s fun to discover those still happen – I highly recommend if you’re in the area). And the time she took us to a ranger talk at a National Park where the ranger offered a postcard to anyone who asked a question he couldn’t answer – she promptly earned all his postcards with questions she genuinely wanted answered. She would stop and talk to keepers at the Philadelphia Zoo to learn more about the animals (a habit my wife and I both now share).

Because of her constant interest in, well just about everything, I have been to historic sites, factory tours, art and science museums, zoos, and many other places. My wife – a curious person herself with her own curious parents – and I continue those kinds of adventures. We go places, we ask questions, and we learn things.

We also read widely, both in our professional fields, and about the world in general. Long drives often involve several hours of listening to podcasts featuring experts in a variety of fields. I’ve previously talked about those as my ongoing Liberal Arts education.

If you aren’t someone who was trained to be curious by your family, it turns out you can do things to increase your curiosity. And yes, I found that article cause I was curious to know if curiosity is trainable – there are deeper articles on curiosity out there too.

The value in being curious is that it causes you to learn things you might otherwise have missed. Some of those things just make you smarter about your world – and the world needs more smart people. Others will suddenly pop up in your work and make you better at your job. They might come up in random conversation with a new friend and deepen your connection to another person.

Learning new things is never a bad idea.

Always Learn More

All this advice boils down to this central idea: always being learning something new. It’s good for your brain and good for your life (oh yeah, and career).

Reading through all of this commentary on professional development you might think: “Wow, that takes a lot of time!” But if you put yourself into a mindset of always trying to learn something new much of that times starts to overlap with other activities in life. And when it’s not all dedicated directly to work it doesn’t burn you out the way constant study of one technology of platform can.

Push yourself forward. Find things that are interesting to you. Nothing new is a waste of your time.

Reminder of the Options

Part 1 provides a detailed review of the directions that data can flow, and the types of architectures you can use to support your solution:

• Data can be outbound to constituents, provided by individual constituents, or exchanged among constituents.
• There are all-in-one solutions, two-system solutions, and solutions with an API or data proxy layer.

Three patterns, three potential solutions: it’s temping to think they align 1:1 – but then a whole second article would have been a waste of time. But it is important to keep them in your mind as you determine the best-fit for your organization and use case(s).

Portal Architecture Considerations

To make the right selection you need to balance your overall data ecosystem, staffing, budget, audience interest, as well as projected growth over time. I keep wanting to create a simple decision matrix, but an easy solution eludes me still.

Audience Engagement

The first major consideration is how will your audience actually engage with the portal. This should drive not only how you build it, but if you build it at all. Lots of organization believe their constituents are all waiting for a great portal to see all their information, update their addresses, and even communicate with each other. This is rarely true. None of us is short on things to do online – you need to be clear about what makes your portal worth their time.

When you align your project to your audiences interest, a portal is powerful, useful, tool that can make a huge impact. But if your audience isn’t interested in what you offer them, you just spent a bunch of money on a web site no one uses. There are times when the most effective constituent portal is none at all – because you can use those resources elsewhere.

Think carefully about how many of you constituents want to log into your portal enough to:

1. Remember it’s there.
2. Care enough to take the time to log in.

You spend something like 40 hours (probably more) a week thinking about your organization’s work. They spend 40 hours (or more) a week thinking about their employer’s work, also their families, friends, all the companies and nonprofits they interact with, politics, meal planning, shopping, and whatever else – there isn’t much time left for your organization.

Think about the features you can provide that are valuable to your audience and the best way to deliver those. For example, organizations often want to give people places to update their address or payment information. But think about the last time you moved, did you want another place to change your address information? Another common ideas is that nonprofits want to make delivery of annual tax receipts cheap and easy. If successful that means a wave of use once a year that might save your on mailing costs – of course you can also ask for a gift in that mailing so it might not actually be a net gain.

Questions to ask:

• What problem does this portal solve for my constituents? Do they care if I solve it?
• What is the best way to deliver our solution to our constituents?
• Will my constituents be willing to log in at all?
• How will this portal empower themselves or to support our work?
• How many people need to use the portal each week, month, or year for it to be successful?
• Do we expect that number to grow over time?
• Will likely traffic be spread out, or come in a wave?

Technology and Data Ecosystem

Your portal does not exist is a vacuum. Your organization has lots of technology kicking around these days, and you want the tools you select for this project to build on your overall technology landscape. When I make a recommendation to a client, I spend a lot of time asking them about the tools that are already in place and how they are used. I’m looking for tools that could be the basis of the portal, reveal the expertise of their staff, and would work well with Salesforce (or other technologies I’m part of introducing). I am also trying to understand some of the history about those tools and how they came to be.

If there are other complex systems, and they are working well, that’s a sign that an organization can probably handle more complex architectures. It’s also a sign they probably know how to benefit from that complexity. But it’s also a sign their staff may be consumed with what they have. If they have all simple systems I am going to be inclined to keep my solution equally simple.

Questions to ask:

• How many interconnected data systems do we have today? Do they work well, or are we struggling to keep up?
• If we add a new system will it enhance our digital presence or pull attention from other critical systems?
• Are the solutions we are considering related to technology we already use, or are they entirely new?
• What kinds of resource, limits, and features does my main CRM provide?
• What other technical resources do we already have in place that we could leverage?
• Did past projects go well, or leave a bad aftertaste?

Staffing

Your solution will require staff support. Even the simplest of systems will trigger requests for help using it. The more complex the solution, the more skilled your staff need to be to make sure it runs smoothly – and the more staff you need to support it.

You staff bring two things to the table: time and skills. They need to have the skills to support your solution, and the time to use those skills effectively.

Please do not just plan to push your staff harder, you need to find ways to remove other tasks from their plate or add people to your staff if you want to avoid burn out.

If you use an all-in-one solution, you probably just need people who understand that platform, know how to engage with your user constituents, and can work with to provider’s support team. The more systems you need the more platforms your team needs to understand, and at least one person needs to have a deep understanding of how the whole thing works.

You also should consider aligning your technology selection to what your team either knows, or wants to know. Most of the time you want to play to your team’s strengths not ask them to develop a whole new skill set from scratch.

Questions to ask:

• What roles do we need to have for your portal to work?
• What work do we want done by our staff vs consultants?
  • Who will answer support requests?
  • Who will respond if there is an outage?
• Do our current staff have time for new responsibilities?
  • How can we reduce their workload to offset new requests?
• Can we hire more staff?
• What technologies do our current technical team members already know?
  • What are they interested in learning?

Budget

Obviously all this will cost money. There will be some combination of license costs, hosting fees, implementation costs, support contracts, staff salaries, and other expenses. Just because a sales person says their solution works “out of the box” does not mean it’s free to get started.

You need to account for the implementation cost: how much will it take to setup and train people on the new system?

You also need to plan for maintenance costs: what will it cost to keep it running every year?

In very broad sweeping terms my experience is that most of the time an all-in-one solution will be cheaper and faster to implement than a two-system solution. But all-in-ones often have higher annual operating costs. Those operating costs are likely to grow linearly with audience size. A two-system solution, while more expensive to setup because of the complexity, will often have a lower annual cost and lower price escalation curve.

When you are considering your options, those are the two budget figures to make sure you understand. What will it cost to setup? How much will it cost over the first 5 years? And what factors will impact those two figures?

Different organization can tolerate different cost structures. If your organization prefers to invest up front and have lower long-term costs, you should make a plan to optimizes for that. Other organizations want the lowest upfront risk, but can tolerate higher costs – particularly if they are linked to a successful and growing solution.

More complex systems are also riskier to build. There is a greater chance of hitting problems that require extra budget to resolve.

Questions to ask:

• What will each of my main choices cost to implement?
• What are the 1, 2, and 5 year costs likely to be?
• What factors will drive costs?
• What is my organization’s budget tolerance for up-front vs long-term costs?
• How firm is the budget? What are my contingency options for overruns?
• What are my options to control costs if there are problems?
• What can our staff realistically do vs what do we need to pay consultants to do?
• If we learn the portal isn’t working, what will it cost to cancel our contracts early?
• If my traffic will be in small bursts, will I have to pay extra for those bursts, or pay all year for services we’re not using?

Closing Thoughts

There is no one right answer to these questions, and they lead to complex matrix of decision making.

Take the questions I’ve posed, and have a robust and honest discussion about them. Ask your own questions. If you have a consulting partner you trust, have them test your answers and add their questions as well (sure, they are incentivized to get your to build it, but they should still have insights you want).

To make your final selection I suggest you write out a 3-5 year plan – this is a thing consultants can help you do well. Test it with your internal team before taking it to leadership. Make sure you understand it, and that your team understands it, particularly if a consultant wrote it.

I’ve been down this road a few times, and seen organizations make great decisions that led to successful projects, careful decisions that led to portals no one used, and wise decisions to walk away from their vision for a new plan. You need to be open to all possible paths.

Good luck!

This is the first in a series on what your choices are, and now to select the right one. The architecture you select will drive the implementation cost, maintenance costs, and scalability.

The Purpose of Your Constituent Portal

Before deciding on the architecture of your portal, you need to have a clear understanding of its intended purpose and expected scale. The purpose of the portal will also dictate the direction(s) data flows, and the scale will help you plan the long term costs.

If you can’t clearly state why you need a constituent portal, you don’t need one.

Different organizations have different reasons to create portals. Companies that sell products may want a warranty or support portal. Nonprofits often want a donor portal that provides tax receipts and allowed recurring donors to update their gift information. Member organizations want a place for members to see their benefits, update their information, and renew their memberships. And so on.

The purpose of your portal will determine the direction that data flows through it. There are essentially three directions that data can move.

Outbound Data

You can send data from your organization to your constituent. That could be things like receipts, memberships status, or product warranty information. Anything that is data you have, that your member might need to see, but not update.

Data Exchange

You can allow constituents to update information. That could be address changes, support requests, donation schedule updates, event registration details, and more.

Data Network

Your third option is to allow you constituents to exchange information with each other. This could be a any form of member community where they engage with each other.

When planning a network, remember you have to plan for content moderation and acceptable use policies.

Primary Portal Data Architectures

To support your portal, whatever the data flow is, there are three main data architectures you can choose from, each which a different strengths and weaknesses. The more complex your architecture the more options you will have for what it can support, but it will also increase costs and maintenance efforts. Bigger is not always better – sometimes it’s just more expensive.

1. All-in-One Constituent Portals

In an all-in-one solution your constituent portal is part of your CRM. For obvious reasons this is the type of portal that your CRM vendor wants to sell you. In Salesforce land this means Experience Cloud. There are also many nonprofit-focused CRMs which include one in their solution. When your portal is part of your CRM you get a bunch of advantages:

• Data all lives in one place. There is no data sync to worry about setting up or maintaining.
• You have one vendor. That means you can centralize your support arrangement and billing.
• They are often fastest to implement. They are designed to be fast to market to help make them the obvious choice.
• They generally do not require a developer. Again, your CRM vendor really wants you to select this path, so they clear as many hurdles as they can.
• There is only one technology stack. By leveraging the technology of your CRM your investments in learning that technology carry over, at least in part, to your portal.

There are also downsides:

• New template system to learn. The CRM’s portal likely has a different template system than the CMS on your main web site. That may be hard to make match your primary online branding.
• The vendors make a lot of assumptions. To provide that ease listed above they make assumptions about how data flows, security, user experience, and how other elements should work – those assumptions may not match your ideal solution.
• Costs often scale linearly. There are usually license costs that are scaled based on user count meaning your costs grow as the portal grows at more or less a 1:1 rate. While there are price breaks and other incentives this is the right expectation to have for estimating.

2. External Constituent Portal

In this pattern an external constituent portal moves the user experience from within your main CRM’s sphere of influence into a separate platform. In my career I’ve mostly built these with Salesforce as the CRM and Drupal as the portal platform – it’s a powerful pairing. The strengths and weaknesses here are more or less the mirror of the all-in-one portal.

• Your web team may already know the technology. If you use the same technology as your web developers use for your main web site(s) they already know who to handle design and branding.
• You have greater control over the User Experience. The assumptions that go into the portal are yours not the ones imposed by the CRM.
• You have greater control over security, along with data flow and formats. Since they aren’t built around assumptions you don’t control you have greater control and freedom.
• Costs typically grow more slowly. It is less common to have per-user license costs in this context so the cost curve is likely closer to logarithmic.

The downsides:

• You have to handle data synchronization. The two systems means data has to pass back and forth.
• You have two different vendors/platforms. When you had one system everything was centralized, now you have two different systems handling constituent data.
• This design typically is harder to implement. All those short cuts your CRM provider had in mind for you are gone. Even if you use a purpose built solution it’s going to take more time and effort.
• This approach generally requires a developer and/or data architect. To implement this pattern you need someone who understands both platforms – ideally both for setup and support.

3. External Constituent Portals with Data Proxy

In some situations it makes sense to insert a data proxy, or API layer, between your main CRM and your constituent portal. This creates a layer of abstraction, security, and data augmentation that can benefit a lot of organizations. While this is the most complex of the three main architectures it is one I find is frequently overlooked – in part, I suspect, because no one partner benefits from selling it.

This setup certainly isn’t for everyone, but when it’s the right fit it makes a huge difference. It also creates a significant number of iterations – all of those arrows could be one directional or two directional.

An extra data layer makes the most sense when data from your CRM is going to multiple places and/or needs to be augmented by an external data source. For example think about an organization that provides trail maps for hiking or paddling. All that data geocoding those trails does not need to be in your CRM, but you probably want to know which members are interested in which trails so some trail data is needed there. Meanwhile in public you want a great deal more information all linked together. An organization like that may have a portal for members, a web site to find trails for non-members, apps for iPhone and Android, and perhaps a public API for other people to use to build their own web sites with. That’s a lot of API calls all with different data sets, all of which need to be very fast.

Your CRM is designed to be very stable and reliable – it is not designed to be very fast. What’s more, you often have limits on the number of API calls you get in your package (with extra fees charged if you go over). By inserting another layer between your CRM and other services you can bypass these limitations.

There is another added benefit to this design pattern, and that is increased security. By creating an additional layer in your system you are able to create a separation of concerns between the various systems. Each should have access to only the data is absolutely needs from any of the others.

How to Pick?

There are lots of considerations that go into selecting the right architecture for your project – which is why that’s getting it’s own post soon.

Chris Zietlow presented back in September about using Machine Learning to Improve UX.

We frequently use these presentations to practice new presentations and test out new ideas. If you want to see a polished version hunt group members out at camps and cons. So if some of the content of these videos seems a bit rough please understand we are all learning all the time and we are open to constructive feedback.

If you would like to join us please check out our up coming events on Meetup for meeting times, locations, and connection information.

Thank you to everyone who worked so hard to make the event a success.

I’ve assembled a playlist of the various sessions I though were good when I attended, or looked good but couldn’t attend.

Monday I attended the Community Summit, and while I had lots of great discussions, I didn’t take a lot of notes. The biggest two things I noted were that Western New York DUG is doing interesting stuff with online meetings that might be worth checking out and emulating for the SC DUG. And that Mid-Camp keeps a list of all the various channels that have videos of Drupal Camp sessions.

DriesNote:

Roadmap: The current roadmap looks pretty cool, assuming everything comes together as well as we all hope it will:

Announcing the new #Drupal8 Roadmap revealed by @Dries during today’s #DriesNote in #DrupalCon Nashville https://t.co/wKCQfvcbAr pic.twitter.com/mK5s4afe71

— DrupalCon Nashville (@DrupalConNA) April 10, 2018

Dries showed off some great stuff from the new demo site called Umami. Umami has been committed for 8.6, and we might be able to see it later in 8.5

Small correction: Umami demo is *already* in #Drupal 8.6 and we are hoping to even expose it in a later release of 8.5! #DrupalCon #driesnote

— webcsillag (@webchick) April 10, 2018

JS modernization and a new admin interface design are on their way, media library is part of that, but is likely a year out from being ready for prime-time.

Webchick summed this section of the talk nicely:

So rad to see the Out-of-the-Box and Layout initiatives being shown off together. Feels like #drupal 8 is really coming together for site builders and content authors! #driesnote #drupalcon

— webcsillag (@webchick) April 10, 2018

We are very over due for the needs of content creators, so it’s great to see meaningful headway on some of these processes.

Dries then moved on to start talking about values. It’s something he’s still not clearly fully comfortable doing, but it was good to see him try. The first public version of his attempt to define a set of values is up.

My read is that its well intended and has some ground to cover is it gets revised. I haven’t done a deep dive into its details yet, nor the response, but early reviews are mixed. https://twitter.com/hrodrig/status/983725468405895169

Although there was much less discussion today in hallways and informal chatting than I’d expected to here.

And there is definitely some ground to cover on issues that got us here in the first place:

https://twitter.com/hanabel/status/983743965047545857

(That’s not related to the DriesNote directly, something she ran into at later session but was on topic of my comments)

JavaScript and Accessibility: Don’t Blame the Language

This was a really good session on accessibility with both a real world set of examples and realistic discussions of what’s hard and what happens when things pass tests but don’t get tested by humans.

Major take aways:

• Modern tools support JS and it no longer gets in the way of accessibility. WCAG 1.0 said this was a problem 20 years ago, but that’s not the current best practice.
• There are constraints to the work because of accessibility, but it they don’t have.
• “There are times that I go to use an interactive calendar on the web and all I hear is 1, 2, 3, 4, 5, 6, 7, 8, 9, and so on to 30 or 31…with no indication that these are dates…just a mass of numbers in the middle of the page.”
• We used to test sites by disabling CSS/JS. Now it makes more sense to try to navigate the site with a keyboard and see what happens. Remember that just because something is possible it doesn’t mean it’s obvious or good. This doesn’t get you to a great site, but allows you to pick off errors before someone finds them for you later.
• When you tab to things, the visual affordances some designers hate can be put back in as a compromise for people using accessibility tools.
• I need to spend more time with the iPhone voice over tool so I can test things better.

Things I didn’t go to in this window:

• Making Inclusion Happen Through Mentoring
• Build banging sites with BPM: Bricks, Paragraphs and Modifiers
• Big Changes for Small Agencies (this one got mixed reviews from a couple attenders).
• Managing Your Most Important Resource: You

UX for Admin:

This was a really interesting session on the Material Admin theme, and what’s been needed to make it work.  It’s not perfect, and may or may not be ready for prime-time, but it looks like a great idea and show what we can do to make the admin much better.

Related projects:

• https://www.drupal.org/project/material_admin
• https://www.drupal.org/project/material_admin_support
• https://www.drupal.org/project/type_style

Major take aways:

• We’re behind, some of fixing that is easy, some of fixing that is hard.
• When you’re UX is bad, people perceive things to be slow even if they aren’t. People think that material theme is faster even though it is demonstrably not.
• Growth and survival of the project require us to have a better admin.
• He’s trying to make sure add-ons for the theme/module are pretty standalone and just work. But theme’s can’t require modules which is silly.
• Contenta uses Material by default on front and back because it provides decoupling well.

Skipped in this window:

• Taking Images to the Next Level (in Drupal 8)
• Web Accessibility in Higher Education (Canceled)
• Drupal Core Auto-Update Architecture
• PDFs in Drupal  (I was surprised that this one was overcrowded)

Salesforce BOF:

This BOF was a chance for Cornell to show off some great stuff they have been doing with Message Agency. They have done some cool stuff that shows the power for D8 and a good Salesforce integration.

https://twitter.com/digitalburns/status/983792403365679104

It should go without saying, but it needs saying too much:

A Salesforce is a CRM. Drupal is a CMS.

Use your tools for what they are best at.

The content in Drupal, actions recorded back into Salesforce.

Lessons:

1. Know strengths of each tool
2. Understand user needs
3. Determine how you will use each tool
4. Get the details right: SSO, Data Mapping, etc.

Drupal is much better at providing accessibility, including Form Assembly which is hard. The SF eco-system is mixed on the whole.

One option for multiple databases is Snap Logic (apparently it is “capital intensive”).

So you have a Code of Conduct… now what?

This was a mini session that is worth watching if you’re unsure about the importance and value of having a code of conduct.  The hope had been to have a discussion about the importance of Drupal’s CoC, but everyone who attended largely agreed about the broad strokes of the major issues that have been discussed lately in the community.  We ended up talking more about how to broaden the discussion than about the CoC itself.

Skipped:

• Keeping Kittens Safe While Altering Drupal Core

Handling a Big Year: ACLU.org in 2017

This session was an interesting look at the impact on ACLU’s D6 (yes that’s right) advocacy site running on Pantheon.

Moved to Pantheon in 2013. And that move dealt with limits of their old hosting solution. Unfortunately some of my old-timey knowledge of why that had that solution was so old they couldn’t tell me much about how they had managed to make that move.

“Crazy things happen all the time”

After the their ED made a Rachel Maddow appearance on 11/16/16 they saw an 85x traffic spike. Tag1 was called in to help sort out what happened.

They found it was database bound, which was very common on D6, but still something they see frequently.

Found queries with 3 table join with no indexes on the base table. Able to go from 200,000 rows being scanned, down to 76. They were responding in real-time in crisis response mode.

After the wave passed, they called Pantheon to help build out environments for testing using multi-dev.

During the spikes that followed for the first travel ban, which were even larger they worked to reroute errors to Fastly, which served a PayPal fundraising link: at least the donations kept coming but that wasn’t good enough.

They needed a botnet to replicate the traffic. Tag1 used: Locust to create load tests, SaltStack to organize the bots, and EC2 to be the bots. They were failing at ~600 requests per minute and they were able to get to ~5,000 requests per minute. At that point the payment gateways were also starting to buckle, which isn’t a thing most people see.

The final wave they discussed came after the Net Neutrality lose, which peaked around 1,900 form submissions/min.

ACLU needed more logging, but didn’t want them logging personal information. Turned out the payment gateway’s CDN was detecting a DDOS and blocking them. See curl_log and curl_loadbalance. They also intentionally shift load from MySQL to Redis and PHP(?!?) because they knew Pantheon could scale that are far and as fast as needed to handle the waves, but MySQL was a limiting factor.

Skipped:

• Automate Your Automation
• Principles of Content Layout

Wednesday Keynote by Steve Francia

Creator of Hugo and BFD in the Go community. DA Board member.

https://twitter.com/ShawnBorton/status/984076235260211202

https://twitter.com/karolus/status/984079263140691968

He has been helping Google put together documents to help guide their engagement with Open Source communities and projects.

https://opensource.google.com/ https://opensource.google.com/docs/

Things he argues we pioneered in Open Source: - Distributed leadership - Collabortive Development - Community Engagement

https://twitter.com/webchick/status/984086341368426496

https://twitter.com/ShawnBorton/status/984086225287041024

Interesting to reflect that its about the process and the community, but not about the technology.

Unrelated:

https://twitter.com/eaton/status/983789427557269504

https://twitter.com/red__cat/status/984104168339267586

BOFs Board:

https://twitter.com/tk61281/status/984095554178138117

Manage Yourself First

80% of leadership is just showing up. The other 20% is actually caring.

You will make mistakes. How you handles those mistakes will define your success.

Find Trusted Mentors

Do what people think you can’t.

Guide teams to successful outcomes.

People buy outcomes, so do teams.

You must be prepared to react positively to the unexpected

You should read books that speak to you, but you should also read books that aren’t about you or people like you.

Ken really likes even though it is written in the most macho way possible: Just Listen: Discover the Secret to Getting Through to Absolutely Anyone

Go back to a point for basic agreement, even if that agreement is basic facts and that you screwed up.

Empathy is a great thing that asks you to choose sides even when you shouldn’t.

Your job is not to normalize and enforce things.

Down Girl: The Logic of Misogyny Mother Knight

Skipped:

• https://events.drupal.org/nashville2018/sessions/dont-trust-your-gut
• https://events.drupal.org/nashville2018/sessions/organizing-wordpress-community-victories-challenges-and-lessons-learned
• https://events.drupal.org/nashville2018/sessions/continuous-integration-has-never-been-so-easy
• https://events.drupal.org/nashville2018/sessions/accessible-editor (https://twitter.com/hanabel/status/984105892911239169)
• https://events.drupal.org/nashville2018/sessions/new-help-system-drupal
• https://events.drupal.org/nashville2018/sessions/enforcing-code-conduct
• https://events.drupal.org/nashville2018/sessions/extending-abstract-class-privilege-outcomes-and-lessons-learned
• https://events.drupal.org/nashville2018/sessions/community-convos-camp-organizing
• https://events.drupal.org/nashville2018/sessions/media-module-core-setting-drupal-8-media-library

Salesforce BOF

Mostly a good discussion and a few new ideas. Also good to catch up with old friends and ideas.

Community Convos: Governance Retrospective

(Note: recording was intentionally stopped after the presentation but the discussion continued for quite a while).

Following Con last year Whitney Hess put forward some ideas, but it wasn’t clear where to go next.

It wasn’t clear that the DA should lead this, so it fell to the CWG cause they were last group standing.

Take aways:

• Governance should evolve over time.
• Need a values statement
• Need to define the community and its membership.
• Clearly document that structures and procedures.
• CWG needs to improve CoC and enforcement.
• Community needs to improve its global outreach.
• DA should set higher standards.
• Community matters should escalate to groups, not individuals.
• We need community onboarding.
• We should engage with other communities to discover best practices.

What’s Happened:

• Dries stepped down as DA board chair.
• DA hired Rachel Lawson.
• DA created an updated CoC.
• Dries is doing a round table on Thursday.

What’s Next:

• Trying to figure that out…
• Need to determine if good feedback was gathered so far.
• Need to figure out an ongoing and continuous feedback process.

The expected frustrations with Dries and the values statement were expressed. Communication between Dries and other folks continues to be a challenge. The bottleneck of single point of contact is making it hard to stop having a single point of contact.

Skipped:

• https://events.drupal.org/nashville2018/sessions/continuous-integration-nirvana-tricks-reach-heavenly-automation
• https://events.drupal.org/nashville2018/sessions/hostile-drupal-tips-tricks-running-drupal-hostile-environments
• Cthulhu Drupal: Coding with Lovecraft

Drupal for Nonprofits BOF

The main discussion centered around what’s holding back D8 adoption and the ongoing sense that the main forces in the Drupal community no longer concern themselves with the nonprofit sector.  This year’s BOF was small because NTC started today in New Orleans.  From a rough head count if the people I new were in New Orleans had been at the BOF there would have been a similar number of people.

Unrelated:

http://www.ashedryden.com/blog/the-ethics-of-unpaid-labor-and-the-oss-community

https://twitter.com/mikeherchel/status/984441643343507456

https://twitter.com/DrupalBDays/status/984444344437477376

Thursday Keynote: Emily Rose aka Amorelandra

https://twitter.com/kattekrab/status/984438962780590080

In 2013 51% of internet traffic was “non-human”. SEO industry calls it NHT. By 2014 it was 61%.

https://twitter.com/sugaroverflow/status/984440175546814465

https://twitter.com/ShawnBorton/status/984443812675190784

Q&A with Dries.

Related Twitter Hastag Automatic’s Support of Camps and staff to do so: It’s great, but it’s not in the budget (DA budget). He talked about creating it as a DA service that could be self-sustaining, but the WordPress model includes a donation of 8 FTEs.

https://twitter.com/uniongal/status/984513830930694144

What if your responsible for 1000 D7 sites? When we will know when there is a concrete answer to the question of the EOL for D7: This is an open issue without a good answer that needs a good answer. Move to D8? (but he doesn’t understand why that’s laughable without more detail).

What about the small shops and builders: He doesn’t feel like they were really left behind. Rachel also checked to what the DA could have done better with the new home page, but the language wasn’t a great choice.

https://twitter.com/rootwork/status/984515473868980224

What can a consumer do to preserve the open web: Not use Facebook. People read the web through Facebook like they do with Google. Don’t install an ad blocker.

Why don’t you hear more about Diversity issues from you? It’s important, and we have to do better. We aren’t were we should be, and I’m happy to show more leadership. I could do more by talking more about it in public and on twitter. Wants to think more about it, and doesn’t feel like an expert. He acknowledged his mistake in the DriesNote in Copanhagen. He also commented about shuttingdown after being called out because of how it was done. Wants understanding of the fact that he’ll make mistakes.

https://twitter.com/rootwork/status/984517468646510592

When are we moving to Github? A proof of concept is in place to move to GitLab! Our tools are better than GitLab in many ways, but GitLab wants to have our better strengths in their code base. So they are working on doing that for us and for all their users.

Is Drupal 7 Dead? No. Most sites are Drupal 7, and some new sites still launch there. But all the innovation is on 8.

https://twitter.com/rootwork/status/984520931283320832

The new values and principles need work to more fully reflect the community. The process: a group together in December to review the community feedback. And it was clear he needed to do this. He’s been working on it since then, and has found it hard work. He wanted to make it Collabortive, but also wanted to put a stake in the ground. He knows that it needs work, but isn’t entirely sure of the next steps. Doesn’t want to the single owner. He would like to assemble a working group with a charter.

https://twitter.com/rootwork/status/984522254049562626

Did you create drupal to be modular and community driven from the start or did it change into that over time?

https://twitter.com/tonylegrone/status/984518038904082433

There were other CMSes in the world, but they were a shit show. I was working on the Linux kernel and liked the modular nature.

Why do you think people are hesitant to update their site? Decided to elect minor updates not major. Mostly that it add complexity.

https://twitter.com/rootwork/status/984523890650243077

Will the new principles state that destructive beliefs, not just actions, will be banned. He defers to the working group.

https://twitter.com/bradrjones/status/984525189819125760

https://twitter.com/drnikki/status/984523437573099520

Rachel acknowledged the tweet, but didn’t know what to do with the fact that it actually called her out. “I wasn’t paying attention.” and then blamed questioners for not asking questions earlier. @drnikki was given a space, and directed people to DD&I meetings.

A really bad response from the audience calling on women lead. Tim Plunket responded appropriately.

https://twitter.com/rootwork/status/984526464690343936

https://twitter.com/eaton/status/984183504207564801

Random Unrelated:

https://twitter.com/webchick/status/984531352321077248

https://twitter.com/geerlingguy/status/984533604687630337

https://twitter.com/fldrupalcamp/status/984533829523333121

What we’re doing here is creating a database service and a controller to provide a JSON endpoint. We’ll define the database connection, the Drupal service, and then the controller.

Drupal allows us to define database connections in the main settings file. This allows easy access to Drupal’s database services and query classes.

Connection Definition

The first step is to define the database connection in settings.php:

$databases['remote']['default'] = [
  'database' => 'extra_data',
  'username' => 'accessingUser',
  'password' => 'UseGoodPasswordsIn2017&Beyond',
  'prefix' => '',
  'host' => '10.10.1.1',
  'port' => '',
  'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
  'driver' => 'mysql',
  // I'll explain this detail later.
  'pdo' => [
    PDO::ATTR_TIMEOUT => 5,
  ],
];

Services

Next we need to create a new database service using the new connection information. Drupal core’s database service can be leveraged to connect to additional databases by just defining your new connection as a service in your module’s services.yml file.

services:
  mydataservice.database:
    class: Drupal\Core\Database\Connection
    factory: 'Drupal\Core\Database\Database::getConnection'
    arguments: ['default', 'remote']

Notice the arguments from the database service are the array keys (in reverse order) from the settings.php definition of the connection.

That’s all it takes to create a new service that wraps around your database.

The Controller

That service is all well and good as far as it goes, but if we want to actually to send the data to the browser we need a controller to leverage our new service and send the response.

Using dependency injection I attached the service to the controller and it looked like it worked great.

<?php
/**
* Constructs a new DataSearchController object.
*/
public function __construct(ConfigFactory $config_factory, Connection $dataservice) {
  $this->config = $config_factory->get('mydataservice.datasettings');
  $this->database = $dataservice;
}

/**
* {@inheritdoc}
*/
public static function create(ContainerInterface $container) {
  return new static(
    $container->get('mydataservice.database')
  );
}

public function getData(Request $request) {
  $query = $this->database->select('mytable', 'mt');
  // From here you gather your data and send your response...
}

In fact it did work flawlessly all the way through initial testing. Using the database service to get the data and the cacheable JSON response technique I’d worked out previously everything came together quickly. You could make a request of the controller with your search terms and the browser gets back a list of objects for display.

Then just before launch the client physically relocated the database server and didn’t tell us it would be offline. Turns out we hadn’t tested what happens to an injected database service when there is no response from the remote database server. The request would wait for the database connection to time out and then throw an exception that didn’t get handled in my code at all. There was no place to add a nice error message and it was incredibly slow since the timeout was 30 seconds.

So at the last minute I had two more problems to solve: trap the error and shorten the timeout on PDO connections.

Drupal 8 database services attempt their connection when the service itself created. It you use dependency injection that means exceptions need to be caught in create(). But create() cannot send a response to the browser, that has to happen later when the function that corresponds to the active route is called by the kernel.

My solution was to make the database service an optional parameter on the controller, and adjust the static returned by create based on the exception thrown:

<?php
/**
* Constructs a new DataSearchController object.
*/
public function __construct(ConfigFactory $config_factory, Connection $dataservice = null) {
  $this->config = $config_factory->get('mydataservice.datasettings');
  $this->database = $dataservice;
}

/**
* {@inheritdoc}
*/
public static function create(ContainerInterface $container) {
  try {
    return new static(
      $container->get('config.factory'),
      $container->get('mydataservice.database')
    );
  }
  catch (Exception $e) {
    return new static(
      $container->get('config.factory')
    );
  }

  public function getData(Request $request) {

  if (!$this->database) {
    throw new HttpException(404, $this->config->get('database_offline_message'));
  }
  $query = $this->database->select('mytable', 'mt');
  // From here you gather your data and send your response...
  }
}

The other way to handle the exception would be to load the connection from Drupal’s service container when you need it in place of using dependency injection for that service.

Also, notice we’re throwing a 404 error. Ideally it would return a 5xx type error, but those trigger other behaviors that prevented me from providing nice errors for the JavaScript application to process easily. Our controller also had a page display (to send the JavaScript libraries and base markup for the actual interface on application startup), which meant that we needed to create a reasonably well themed response in that function as well:

<?php
// If the database is offline, then send error message.
if (!$this->database) {
  Drupal::service('page_cache_kill_switch')->trigger();
  $message = $this->config->get('database_offline_message');

  $error = [
    '#theme' => 'dataservice_error_page',
    '#attributes' => [
      'class' => ['dataservice', 'database-offline'],
      'id' => 'dataservice-error',
    ],
    '#message' => [
      '#type' => 'processed_text',
      '#text' => $message['value'],
      '#format' => $message['format'],
      '#filter_types_to_skip' => [],
    ],
    '#title' => $this->t('Database Offline'),
  ];

  return $error;
}

Settings revisited

So that fixed the errors, but still meant we had a really long wait for the database connection to time out before the connection error is even thrown in the first place. And now we come back to that PDO section of the database connection definition.

$databases['remote']['default'] = [
  'database' => 'extra_data',
  'username' => 'accessingUser',
  'password' => 'UseGoodPasswordsIn2017&Beyond',
  'prefix' => '',
  'host' => '255.255.255.255',
  'port' => '',
  'namespace' => 'Drupal\Core\Database\Driver\mysql',
  'driver' => 'mysql',
  // Now is when I explain this
  'pdo' => [
    PDO::ATTR_TIMEOUT => 5,
  ],
];

PDO expects you to override settings at connection time not in a settings file. So I couldn’t just update my PHP.ini and call it a day but I also couldn’t find any documentation on how to change any PDO settings. Leveraging the power of open source I started to follow the code paths, actually reading through how Drupal establishes MySQL connections and, found I it.

If you add a subarray in your connection definition keyed to ‘pdo’ Drupal will load and apply those settings instead of the defaults. There are a couple settings that Drupal insists on being right about for performance, stability, and security reasons, but timeout and many others are fair game.

When I first built the site I hadn’t yet gotten my head around Drupal 8 caching, and so the JSON responses weren’t cached and therefore the page was slow. After some issues with the site caused me to have to look at this part of the project again I decided it was time to try to do something about this.

Drupal 8’s use of Symfony means we have great tools for simple tasks like providing JSON responses. Originally in the controller all I had to do was provide the JsonResponse class an array of data and it would handle the rest:

That’s all well and good if you don’t want your response to ever be cached, but Drupal provides a CacheableJsonResponse class that links up to the rest of the Drupal 8 caching engine to provide much better performance than a stand Symfony JsonResponse. But it turns out the docs kinda suck for explaining how to use it. After a great deal of digging I found this Question on StackExchange which gave me what I needed.

Edit: You must enable the Internal Dynamic Page Cache module to get CacheableJsonResponse to work correctly. Thank you for the correction.

Basically this is a dump of links, pictures from various things, and a few stray thoughts. I’ll edit as the week progresses and probably add more thoughts and ideas.

Prenote: https://events.drupal.org/baltimore2017/balti-more-prenote-balti-most-fun-drupalcon (unmitigated silliness)

DriesNote: https://events.drupal.org/baltimore2017/driesnote

Import things to call out:

• Claims we’re launching 15,000 D8 sites per month!
• Field Layout module (experimental in Drupal 8.3 core) appears to be DS in core: https://www.drupal.org/node/2795833
• BigPipe is ready for production.
• Quick edit can do drag and drop image upload.

Greg Anderson’s PHP OSS Workflow tools: https://events.drupal.org/baltimore2017/sessions/development-workflow-tools-open-source-php-libraries

• 

  Some time is more important than others, like outages.
• To find Drupal plugins on Packages: https://packagist.org/search/?type=drupal-drush
• poser.pugx.org provides badges on packagist pages.
• https://scrutinizer-ci.com/ “Sometimes any static analysis tools will give you answers you don’t like so just ignore it.”
• https://www.versioneye.com/ uses the licenses from composer.json to check for both out of date and for compatible licenses. Not terribly useful internally, but user for open projects since you can link a badge to the result. Alternative `composer licenses` which is actually smarter.
• Sami: A symfony component that does a JavaDoc like project for API documents. https://github.com/FriendsOfPHP/Sami

Baby Steps, Lessons Learned & Big plans for Drupal Diversity and Inclusion: https://events.drupal.org/baltimore2017/sessions/year-diversity-initiatives (Core conversations style 30 minute talk with 30 minutes questions).

• https://drupaldiversity.github.io/ For main materials of the group.
• https://drupaldiversity.github.io/resources/ For 3rd party resources to help people get up to speed.

Long line for questions & a lot of conversations at the @drupaldiversity session! 🙌🏻🎊 #DrupalCon @DrupalConNA pic.twitter.com/oQN5wdaWwP

— Fatima Sarah Khalid (@ksFatima) April 25, 2017

Launching Online Stores with Commerce 2.x on Drupal 8 https://events.drupal.org/baltimore2017/sessions/launching-online-stores-commerce-2x-drupal-8

• Main Take away: Commerce in D8 is much closer to ready than I thought, mainly cause like Drupal core much more functionality was moved from supporting modules into Commerce core.
• https://drupalcommerce.org/blog/44305/commerce-20-alpha4-released
• https://www.orlowatches.com/ Is live on D8 Commerce 2
• https://www.drupal.org/project/commerce_stripe Commerce Guys maintained.  Although Braintree is the implementation they seem to like best.

Drupal is Changing, Quickly: How and Why https://events.drupal.org/baltimore2017/sessions/drupal-changing-quickly-how-and-why

• Main Take Aways:  It seems to be working, and we’re confident enough about the current plan that I think it’s a bit more real than it’s been in the past. And there are ways to track (engage in) what’s being considered for new features.
• https://www.drupal.org/project/ideas
• https://contribkanban.com/board/ideas

Composer Resources:

• Jeff Geerling’s tips for D8 in composer:  https://www.jeffgeerling.com/blog/2017/tips-managing-drupal-8-projects-composer
• I expect more notes from the BoF today here: https://www.jeffgeerling.com/blog/2017/composer-bof-drupalcon-baltimore or https://events.drupal.org/baltimore2017/bofs/managing-drupal-sites-composer
• https://events.drupal.org/baltimore2017/sessions/improving-your-drupal-8-development-workflow
• https://www.drupal.org/project/config_installer
• The general sense is that we all clearly need to master Composer.  Drupal core support for composer workflows is weak by default and needs to be fixed.
• https://www.drupal.org/node/2002304
• https://www.drupal.org/node/2768413
• https://www.drupal.org/node/2867757
• List of current issues: https://www.drupal.org/project/issues/drupal?text=Composer&status=Open&priorities=All&categories=All&version=All&component=All&order=last_comment_timestamp&sort=desc

Thanks @kanopi_studios for bringing me to #DrupalCon Baltimore, which has been awesome! Love working & playing with you all! #DrupalThanks pic.twitter.com/7UFc7wJRl5

— Darlyne Dolap (@DarlyneDolap) April 26, 2017

Project Estimates:

• https://events.drupal.org/baltimore2017/sessions/how-changing-our-estimation-process-took-our-project-endgame-wtf-ftw
• Slides: https://drive.google.com/open?id=0B_ISJ0Qb3k1Od1ZIV2FXWTJnLUk
• Outlines a project process for fixed budget projects with two defined points to refine estimates and scope to help achieve success.

“Nobody knows what they’re doing.. they’re just contributing what they know” @webchick #drupalcon

— Heather Rodriguez (@hrodrig) April 26, 2017

Technology and its workforce at ethics crossroad https://events.drupal.org/baltimore2017/keynote-technology-and-its-workforce-ethics-crossroad

I’m gonna need to watch this one again. Love the talk, @zeynep #DCZeynep #DrupalCon

— tara k (@sparklingrobots) April 27, 2017

• Humans don’t panic properly! We panic too late instead of when we can do something about it.

• Programmers shouldn’t trust themselves since they don’t know what will happen with their work later. @zeynep keynote #drupalcon

• Everything is multi-causal.

• Toolmakers’ ideals don’t rule their tools.

  Programmers shouldn’t trust themselves since they don’t know what will happen with their work later. @zeynep keynote #drupalcon

  — Aaron Crosman (@acrosman) April 27, 2017

• Surveillance is Baked into Everything.

• Dismantling structures of accountability

• Labor Realities of New Economy

. @zeynep: The labor realities of the new economy are not compatible with a democracy supported by a middle-class. #DrupalCon pic.twitter.com/yuqivM0CzU

— Eileen Clancy (@clancynewyork) April 27, 2017

Raising the bar with guardr https://events.drupal.org/baltimore2017/sessions/raising-security-bar-guardr

• Guardr provides a bunch of modules and config to help encourage best security practices.
• https://www.drupal.org/project/guardr
• http://www.techrepublic.com/blog/it-security/the-cia-triad/
• https://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project

Watch later:

• Complete Play list: https://www.youtube.com/playlist?list=PLpeDXSh4nHjRbDdwHEBRHItfnjrJ8kEDK
• https://events.drupal.org/baltimore2017/sessions/rescue-me-recovering-sad-broken-drupal
• https://events.drupal.org/baltimore2017/sessions/automatic-drupal-updates-using-visual-regression-continious-integration
• https://events.drupal.org/baltimore2017/sessions/code-standards-its-okay-be-yourself-write-your-code-everyone-else
• https://events.drupal.org/baltimore2017/sessions/compose-your-drupal-website
• https://events.drupal.org/baltimore2017/sessions/we-dropped-security-support-drupal-6-2016-what-happened-after-what-can-we
• https://events.drupal.org/baltimore2017/sessions/%C2%A1viva-la-revoluci%C3%B3n-how-start-devops-transformation-your-workplace
• https://events.drupal.org/baltimore2017/sessions/rearchitecting-drupal-8
• https://events.drupal.org/baltimore2017/sessions/security-emerging-threats
• https://events.drupal.org/baltimore2017/sessions/100-ways-you-can-be-ally

This is what a Drupalist looks like! #humansofdrupal #wearedrupal #drupalcon pic.twitter.com/CE8eFfXn8j

— Diversity&Inclusion (@drupaldiversity) April 27, 2017

Additional Resources to checkout:

• http://technosociology.org/
• http://a11yproject.com/
• https://www.drupal.org/project/gesso

Many hundreds of fabulous photos of DrupalCon Baltimore taken by our amazing volunteer photographic team https://t.co/zN6q6aBOjr #DrupalCon pic.twitter.com/J32TmX2lx6

— DrupalCon Baltimore (@DrupalConNA) April 27, 2017

When I started to work on Drupal 8 I wanted to develop a similar pattern to help create simple conventions for integration with front end work on a project. While the new plugin system helps avoid the ugliness of old hook_block_view() implementations, how to create a twig file for your custom block isn’t obvious and doesn’t impose a naming convention. Worse, I’ve seen lots of example code with blocks that return render arrays of type markup meaning they have HTML in strings in PHP.

Most projects involve a collection of nearly static blocks that provide basic information like the copyright information, a disclaimer text, a link to the firm that built the site, decorative flourishes, and other similar elements that don’t benefit from being managed as content. After a couple experiments I’ve come up with a solution that works pretty well:

1. Create a module for simple blocks (although I use the rest of the pattern anytime I’m creating a custom block).
2. Create the block class for each block within the module at src/Plugin/Block/CopyrightBlock.php.
3. Create a theme function for each block in the main .module file.
4. Create a twig file to implement the theme function within the module at templates/custom-blocks-copyright.html.twig.

The for a copyright block the class itself is simple.  We implement the variables we want rendered by the block, and set the cache to expire at midnight (one of the places a timed cache makes more sense that cache tags):

With the block created, we need to define the custom_blocks_copyright theme function we included in the block using hook_theme() from within the dot module file.  Remember any custom variables you used in the block class need to also be defined here (in this case attributes and year):

Finally, create the twig file to provide the actual markup.

Drupal 8 brings forward lots of concepts from Drupal 7 (and prior versions). It also brings forward a few standard community realities.

The information architecture tools are basically the same at least at the conceptual level. Nodes, Fields, Taxonomy, and Menus all still exist more or less as we’ve known them for years, and while there are differences on the surface those differences are incremental in nature and scope.

We also still have the constant hunt for modules that do what you’re doing so you don’t reinvent the wheel. Many modules that people are used to using are still missing, but new things arrive daily, and many of the new versions are significant improvements over previous generations of tools.

And there is still a lack of a clear line between front-end and back-end. When does business logic end and interface begin? When does a themer need to understand Drupal’s HTML generation vs when does a backend developer need to figure out how to force Drupal to generate carefully crafted markup?  There are opportunities to form better and clearer lines, but they aren’t automatic by any means: every team will have to solve this problem their own way.

What’s so great?

Drupal 8 opens up a collection of new tools and opportunities for the community.

• As a backend developer you get to write modern code. The name spacing can feel a little Java-esc at times, but the ability to properly name space code, ditch globals, move from hooks to event listeners, and other basic OOP tools is incredibly nice.
• With CKEditor in core we get better integration between that interface and the rest of Drupal. And better modules are coming out all the time to solve long standing UX annoyances. For example with the D8 Editor File upload module files and images can both be handled as Drupal file objects, but the editor can know the difference between a file (which should just be a link) and an image (which you should display).
• The Symfony community provides a large number of packages that provide 3rd party integrations, or the tools to make them easy to build.
• The two core base themes do not require keelhauling to make viable. If you commonly built your themes from scratch, the ability to have clean default markup that’s easy to override makes both Stable and Classy a major improvement to life.

Things you need to survive

As you dive into your first projects you need to understand that much of what you know at the detail level has changed.  And so you’ll need to learn a few new tricks and be willing to toss aside a few old ideas:

Your old build standards are wrong. Probably not entirely, but you’ll need new modules in your default builds, new best practices about when to use a node vs block vs custom entities, and other basic details you probably have really well develop standards (or at least habits) from years of working with Drupal.

If you’ve been avoiding it to-date, it’s time to develop an understanding about composer, drush, and Drupal console. Not only do you need to be using these tools (probably all three) but you need to understand what each provides and which tool is best for which job.

Nginx is not for the faint of heart: as best I can find, no one has published a complete setup guide yet. In Drupal 6 and 7 there were pretty good guides to setting up Nginx properly, but with Drupal 8 there are enough differences that those guides don’t really work. And all the guides that I have read so far include errors (some of them significant security mistakes). To be clear, it can be done, but if you want to do it yourself be prepared to do a great deal of extra research along the way.

If you have windows you have pain. There are a number of windows specific challenges (speed and NTFS weakness being the biggest we face), and there is little community support to help you overcome the challenges.

Core patches tend to be required. Unlike Drupal 7, core and module patching is the norm not the exception. There are several issues that are frustratingly slow to get fixed that hit some fairly common use cases ( like this menu block bug). While this is improving all the time, we haven’t launched a D8 site without at least one patch in place.

Cache tags are great, but require learning. The new caching system is powerful, flexible, and totally different from what we had before.  It’s better, but to use it well you’ll need to spend some time getting to know when to set and clear the tags you want to create.

Twig is great, but requires learning and discipline. I really like twig and the much cleaner syntax to brings to the theme layer.  However, as more and more people use it they are finding ways to move increasingly complex logic out of modules and theme PHP into template files. Please fight this urge! Keep your business logic separate from your display logic. If some object wasn’t loaded into a variable in your twig file do not attempt to load it in twig. If you need some 4 or 5 layer selector to get to the value you want to print: fix that in a preprocess function.

The API improvements are coming all the time and make things interesting. So far the community has stayed on schedule of rolling out new minor versions of 8 and that’s meant great new features in each version. It has also meant that sometimes a solution you built is not using the best techniques given the improvements.  That’s not really a problem, but can add headaches in maintenance cycles. Also you will find places where brand new D8 tools are already deprecated but the replacements don’t have good example implementations yet.

XML to JSON in Five lines

In closing I want to share the feature that caused me to realize that Symfony was worth the cost of admission: a 5 line XML to JSON AJAX callback.

My first site had a location finder that links to a data provider partner locations nationwide. That provider has an XML-base API that cannot be accessed directly from browsers, and therefore needed to be proxied through the main site. The full details involve more than just the following lines of code (providing input checking, settings, routing, etc), but the heart of the process is just five lines.

    $client = new Client(); // 1: Create Guzzle Client
    try{
      $res = $client->get($config->get('api_endpoint'), // 2: Make request
            [
              'http_errors'=>false,
              'query' =>; [
                 'key'=> $config->get('coop_api_key'),
                     // ...
               ],
            ]);
      $decoderRing = new XmlEncoder(); // 3: Create XML Encoder
      $xml = $decoderRing->decode($res->getBody()); // 4: Decode the XML Response into an array.
      return new JsonResponse($xml); // 5: Return the array as JSON
    } catch (RequestException $e) {
      throw new HttpException($this->t('Unable to process request'));
    }

New releases of Drupal mean two things: new features and cool new tools, and the retirement of an old version. We got the power and flexibility of Symfony and Drupal 6 sites are no longer getting community support.

Unlike Wordpress, which has well defined upgrade paths, each version of Drupal is a new adventure in upgrade pain. The more I watch people suffer with this pain, and the more I watch them try to find a way to do upgrades that preserve their site’s fundamental structure, the more I come to the conclusion that this pain is telling us something: we’re doing it wrong. Not because Drupal’s strategy is wrong, but because keeping all your content in the same structures is usually wrong. Drupal 8 should not make it easy for you to continue to use an old strategy, it should encourage us to update old assumptions.

Here is how I encourage everyone to view their choices:

If you have a Drupal 6 (or older) site you should update right now. Drupal 6 is no longer getting security updates so you are on borrowed time. But more importantly Drupal 8 is a better tool for the current state of the web than your Drupal 6 site. Most sites running on D6 reflect an online communications strategy that’s at least 4 or 5 years old. Those sites probably aren’t responsive, aren’t prepared to support apps, don’t have the right focus on social media and user engagement, and make assumptions about user behaviors that have evolved. Skip to Drupal 8: do not migrate these sites to Drupal 7.

If there is a tool that is missing from Drupal 8 that your current site uses make sure you need it before complaining (or paying to have someone port it for you). Maybe that tool hasn’t been ported because it doesn’t make sense anymore. Some things are still missing, but lots of things are being rewritten differently because we have a better platform. The community is smarter than it was 5 or 10 years ago, and the platform is better, take the time to figure out why something hasn’t been ported: is it just no one has bothered, or has something better been built instead?

If you have a Drupal 7 site you should update when your web site no longer supports your work.

This is actually the same advice I just gave, but without a few assumptions like “you need a secure site.” Many Drupal 7 sites have a lot of life left in them. A site you built today will be designed to meet the needs you have now, and the ones you foresee in the near future. Three years from now (when Drupal 7 is scheduled to lose support from the community) you will be operating on assumptions that have probably been wrong for at least two years.

Every six months you should ask yourself: does my site reflect my online strategy, and is my strategy still working? If the answer is yes to both of those questions you are fine, if the answer is no to either – particularly the second – you should engage someone to help you update your strategy and rebuild your site.I’ve been part of projects that failed in part because we tried to port a stale strategy and stale content to a fresh site. We broke the new site before it even launched. Don’t try to make Drupal 8 behave like your old site: embrace the change.